Oct 30 04:41:21.108040 ip-10-0-135-98 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 30 04:41:21.108048 ip-10-0-135-98 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 30 04:41:21.108055 ip-10-0-135-98 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 30 04:41:21.108298 ip-10-0-135-98 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 30 04:41:31.345312 ip-10-0-135-98 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 30 04:41:31.345328 ip-10-0-135-98 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 7b07d1f0e52b4d8ca7f103975f01a029 --
Oct 30 04:43:17.639026 ip-10-0-135-98 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 04:43:18.070464 ip-10-0-135-98 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 04:43:18.070464 ip-10-0-135-98 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 04:43:18.070464 ip-10-0-135-98 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 04:43:18.070464 ip-10-0-135-98 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 04:43:18.076185 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076105    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 04:43:18.076185 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076185    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 04:43:18.076259 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076190    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 04:43:18.076259 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076193    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 04:43:18.076259 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076196    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 04:43:18.076259 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076198    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 04:43:18.076259 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076201    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 04:43:18.076259 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076203    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 04:43:18.076259 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076206    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 04:43:18.076259 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076209    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 04:43:18.076259 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076211    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 04:43:18.076259 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076214    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 04:43:18.076259 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076217    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 04:43:18.076259 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076220    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 04:43:18.076259 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076222    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 04:43:18.076259 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076224    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 04:43:18.076259 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076227    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 04:43:18.076259 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076235    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 04:43:18.076259 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076238    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 04:43:18.076259 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076241    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 04:43:18.076259 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076245    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 04:43:18.076259 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076248    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 04:43:18.076728 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076250    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 04:43:18.076728 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076253    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 04:43:18.076728 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076256    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 04:43:18.076728 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076258    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 04:43:18.076728 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076261    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 04:43:18.076728 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076265    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 04:43:18.076728 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076270    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 04:43:18.076728 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076273    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 04:43:18.076728 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076276    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 04:43:18.076728 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076278    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 04:43:18.076728 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076281    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 04:43:18.076728 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076284    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 04:43:18.076728 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076286    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 04:43:18.076728 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076289    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 04:43:18.076728 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076291    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 04:43:18.076728 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076294    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 04:43:18.076728 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076296    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 04:43:18.076728 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076299    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 04:43:18.076728 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076301    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 04:43:18.077191 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076304    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 04:43:18.077191 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076307    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 04:43:18.077191 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076309    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 04:43:18.077191 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076312    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 04:43:18.077191 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076314    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 04:43:18.077191 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076317    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 04:43:18.077191 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076319    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 04:43:18.077191 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076322    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 04:43:18.077191 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076325    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 04:43:18.077191 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076327    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 04:43:18.077191 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076330    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 04:43:18.077191 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076332    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 04:43:18.077191 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076335    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 04:43:18.077191 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076338    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 04:43:18.077191 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076341    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 04:43:18.077191 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076343    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 04:43:18.077191 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076346    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 04:43:18.077191 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076348    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 04:43:18.077191 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076351    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 04:43:18.077634 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076354    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 04:43:18.077634 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076356    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 04:43:18.077634 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076359    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 04:43:18.077634 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076361    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 04:43:18.077634 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076364    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 04:43:18.077634 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076366    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 04:43:18.077634 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076369    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 04:43:18.077634 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076374    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 04:43:18.077634 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076377    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 04:43:18.077634 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076380    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 04:43:18.077634 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076383    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 04:43:18.077634 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076385    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 04:43:18.077634 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076388    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 04:43:18.077634 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076390    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 04:43:18.077634 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076393    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 04:43:18.077634 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076396    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 04:43:18.077634 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076398    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 04:43:18.077634 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076401    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 04:43:18.077634 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076403    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 04:43:18.078075 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076406    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 04:43:18.078075 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076409    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 04:43:18.078075 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076412    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 04:43:18.078075 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076415    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 04:43:18.078075 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076417    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 04:43:18.078075 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076420    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 04:43:18.078075 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076423    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 04:43:18.078075 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076425    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 04:43:18.078075 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076428    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 04:43:18.078075 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076431    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 04:43:18.078075 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076433    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 04:43:18.078075 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076436    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 04:43:18.078075 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076439    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 04:43:18.078075 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076441    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 04:43:18.078075 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076444    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 04:43:18.078075 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076447    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 04:43:18.078075 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076449    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 04:43:18.078075 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076452    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 04:43:18.078075 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076454    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 04:43:18.078075 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076457    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 04:43:18.078583 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076459    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 04:43:18.078583 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076462    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 04:43:18.078583 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076464    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 04:43:18.078583 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076467    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 04:43:18.078583 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076470    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 04:43:18.078583 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076473    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 04:43:18.078583 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.076475    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 04:43:18.078583 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077243    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 04:43:18.078583 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077251    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 04:43:18.078583 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077254    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 04:43:18.078583 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077257    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 04:43:18.078583 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077260    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 04:43:18.078583 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077263    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 04:43:18.078583 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077266    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 04:43:18.078583 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077269    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 04:43:18.078583 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077272    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 04:43:18.078583 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077276    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 04:43:18.078583 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077281    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 04:43:18.078583 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077284    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 04:43:18.079060 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077287    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 04:43:18.079060 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077290    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 04:43:18.079060 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077293    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 04:43:18.079060 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077296    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 04:43:18.079060 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077299    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 04:43:18.079060 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077302    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 04:43:18.079060 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077304    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 04:43:18.079060 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077307    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 04:43:18.079060 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077309    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 04:43:18.079060 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077312    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 04:43:18.079060 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077314    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 04:43:18.079060 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077317    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 04:43:18.079060 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077319    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 04:43:18.079060 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077322    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 04:43:18.079060 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077325    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 04:43:18.079060 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077327    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 04:43:18.079060 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077330    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 04:43:18.079060 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077332    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 04:43:18.079060 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077335    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 04:43:18.079524 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077338    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 04:43:18.079524 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077340    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 04:43:18.079524 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077343    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 04:43:18.079524 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077346    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 04:43:18.079524 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077348    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 04:43:18.079524 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077351    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 04:43:18.079524 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077354    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 04:43:18.079524 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077357    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 04:43:18.079524 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077359    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 04:43:18.079524 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077362    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 04:43:18.079524 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077365    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 04:43:18.079524 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077368    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 04:43:18.079524 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077370    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 04:43:18.079524 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077373    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 04:43:18.079524 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077376    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 04:43:18.079524 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077378    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 04:43:18.079524 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077381    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 04:43:18.079524 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077383    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 04:43:18.079524 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077386    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 04:43:18.079524 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077389    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 04:43:18.080019 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077391    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 04:43:18.080019 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077394    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 04:43:18.080019 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077396    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 04:43:18.080019 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077399    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 04:43:18.080019 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077401    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 04:43:18.080019 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077404    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 04:43:18.080019 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077406    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 04:43:18.080019 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077409    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 04:43:18.080019 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077411    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 04:43:18.080019 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077414    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 04:43:18.080019 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077416    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 04:43:18.080019 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077419    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 04:43:18.080019 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077422    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 04:43:18.080019 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077424    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 04:43:18.080019 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077427    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 04:43:18.080019 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077429    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 04:43:18.080019 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077432    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 04:43:18.080019 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077435    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 04:43:18.080019 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077438    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 04:43:18.080472 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077441    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 04:43:18.080472 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077444    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 04:43:18.080472 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077447    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 04:43:18.080472 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077450    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 04:43:18.080472 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077452    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 04:43:18.080472 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077454    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 04:43:18.080472 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077457    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 04:43:18.080472 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077460    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 04:43:18.080472 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077464    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 04:43:18.080472 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077467    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 04:43:18.080472 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077470    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 04:43:18.080472 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077473    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 04:43:18.080472 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077476    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 04:43:18.080472 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077479    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 04:43:18.080472 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077481    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 04:43:18.080472 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077484    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 04:43:18.080472 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077486    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 04:43:18.080472 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077489    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 04:43:18.080472 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077492    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 04:43:18.080938 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077494    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 04:43:18.080938 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077497    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 04:43:18.080938 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077499    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 04:43:18.080938 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077501    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 04:43:18.080938 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077504    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 04:43:18.080938 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077507    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 04:43:18.080938 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077510    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 04:43:18.080938 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077513    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 04:43:18.080938 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077515    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 04:43:18.080938 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077518    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 04:43:18.080938 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077520    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 04:43:18.080938 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077523    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 04:43:18.080938 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077526    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 04:43:18.080938 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077531    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 04:43:18.080938 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077533    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 04:43:18.080938 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077536    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 04:43:18.080938 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.077538    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 04:43:18.080938 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078038    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 04:43:18.080938 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078043    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 04:43:18.080938 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078046    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 04:43:18.081404 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078049    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 04:43:18.081404 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078051    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 04:43:18.081404 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078055    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 04:43:18.081404 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078060    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 04:43:18.081404 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078063    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 04:43:18.081404 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078066    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 04:43:18.081404 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078069    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 04:43:18.081404 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078071    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 04:43:18.081404 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078074    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 04:43:18.081404 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078076    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 04:43:18.081404 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078079    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 04:43:18.081404 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078082    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 04:43:18.081404 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078085    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 04:43:18.081404 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078089    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 04:43:18.081404 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078093    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 04:43:18.081404 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078097    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 04:43:18.081404 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078101    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 04:43:18.081404 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078104    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 04:43:18.081404 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078108    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 04:43:18.081873 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078111    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 04:43:18.081873 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078113    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 04:43:18.081873 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078117    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 04:43:18.081873 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078119    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 04:43:18.081873 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078122    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 04:43:18.081873 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078125    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 04:43:18.081873 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078128    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 04:43:18.081873 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078131    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 04:43:18.081873 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078133    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 04:43:18.081873 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078136    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 04:43:18.081873 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078138    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 04:43:18.081873 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078141    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 04:43:18.081873 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078143    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 04:43:18.081873 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078145    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 04:43:18.081873 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078148    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 04:43:18.081873 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078150    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 04:43:18.081873 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078153    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 04:43:18.081873 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078155    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 04:43:18.081873 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078158    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 04:43:18.081873 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078160    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 04:43:18.082377 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078163    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 04:43:18.082377 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078165    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 04:43:18.082377 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078168    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 04:43:18.082377 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078170    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 04:43:18.082377 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078173    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 04:43:18.082377 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078175    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 04:43:18.082377 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078178    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 04:43:18.082377 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078180    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 04:43:18.082377 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078182    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 04:43:18.082377 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078187    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 04:43:18.082377 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078189    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 04:43:18.082377 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078192    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 04:43:18.082377 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078194    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 04:43:18.082377 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078197    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 04:43:18.082377 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078199    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 04:43:18.082377 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078202    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 04:43:18.082377 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078204    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 04:43:18.082377 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078207    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 04:43:18.082377 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078210    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 04:43:18.082377 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078213    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 04:43:18.082854 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078216    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 04:43:18.082854 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078218    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 04:43:18.082854 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078221    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 04:43:18.082854 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078223    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 04:43:18.082854 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078227    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 04:43:18.082854 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078229    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 04:43:18.082854 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078232    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 04:43:18.082854 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078234    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 04:43:18.082854 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078237    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 04:43:18.082854 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078239    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 04:43:18.082854 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078241    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 04:43:18.082854 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078244    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 04:43:18.082854 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078246    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 04:43:18.082854 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078249    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 04:43:18.082854 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078251    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 04:43:18.082854 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078254    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 04:43:18.082854 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078257    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 04:43:18.082854 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078259    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 04:43:18.082854 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078261    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 04:43:18.082854 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078264    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 04:43:18.083338 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078267    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 04:43:18.083338 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078270    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 04:43:18.083338 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078273    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 04:43:18.083338 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078275    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 04:43:18.083338 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078278    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 04:43:18.083338 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078281    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 04:43:18.083338 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078284    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 04:43:18.083338 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078286    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 04:43:18.083338 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078289    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 04:43:18.083338 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078291    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 04:43:18.083338 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078295    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 04:43:18.083338 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078297    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 04:43:18.083338 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078300    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 04:43:18.083338 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078302    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 04:43:18.083338 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078305    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 04:43:18.083338 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078307    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 04:43:18.083338 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078310    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 04:43:18.083338 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078312    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 04:43:18.083338 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078315    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 04:43:18.083836 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078317    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 04:43:18.083836 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078320    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 04:43:18.083836 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078323    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 04:43:18.083836 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078325    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 04:43:18.083836 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.078328    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 04:43:18.083836 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079411    2579 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 04:43:18.083836 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079425    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 04:43:18.083836 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079433    2579 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 04:43:18.083836 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079438    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 04:43:18.083836 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079443    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 04:43:18.083836 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079446    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 04:43:18.083836 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079450    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 04:43:18.083836 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079459    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 04:43:18.083836 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079462    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 04:43:18.083836 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079466    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 04:43:18.083836 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079470    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 04:43:18.083836 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079473    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 04:43:18.083836 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079476    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 04:43:18.083836 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079479    2579 flags.go:64] FLAG: --cgroup-root=""
Oct 30 04:43:18.083836 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079482    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 04:43:18.083836 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079485    2579 flags.go:64] FLAG: --client-ca-file=""
Oct 30 04:43:18.083836 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079488    2579 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 04:43:18.083836 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079492    2579 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 04:43:18.084390 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079496    2579 flags.go:64] FLAG: --cluster-domain=""
Oct 30 04:43:18.084390 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079499    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 04:43:18.084390 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079503    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 04:43:18.084390 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079506    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 04:43:18.084390 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079509    2579 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 04:43:18.084390 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079513    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 04:43:18.084390 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079516    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 04:43:18.084390 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079519    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 04:43:18.084390 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079523    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 04:43:18.084390 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079526    2579 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 04:43:18.084390 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079529    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 04:43:18.084390 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079532    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 04:43:18.084390 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079536    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 04:43:18.084390 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079539    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 04:43:18.084390 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079543    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 04:43:18.084390 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079546    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 04:43:18.084390 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079549    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 04:43:18.084390 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079552    2579 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 04:43:18.084390 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079555    2579 flags.go:64] FLAG: --enable-server="true"
Oct 30 04:43:18.084390 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079558    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 04:43:18.084390 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079566    2579 flags.go:64] FLAG: --event-burst="100"
Oct 30 04:43:18.084390 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079569    2579 flags.go:64] FLAG: --event-qps="50"
Oct 30 04:43:18.084390 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079572    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 04:43:18.084390 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079575    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 04:43:18.085005 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079579    2579 flags.go:64] FLAG: --eviction-hard=""
Oct 30 04:43:18.085005 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079585    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 04:43:18.085005 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079589    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 04:43:18.085005 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079592    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 04:43:18.085005 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079595    2579 flags.go:64] FLAG: --eviction-soft=""
Oct 30 04:43:18.085005 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079598    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 04:43:18.085005 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079601    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 04:43:18.085005 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079604    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 04:43:18.085005 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079607    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 04:43:18.085005 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079610    2579 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 04:43:18.085005 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079614    2579 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 04:43:18.085005 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079617    2579 flags.go:64] FLAG: --feature-gates=""
Oct 30 04:43:18.085005 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079622    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 04:43:18.085005 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079625    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 04:43:18.085005 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079628    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 04:43:18.085005 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079631    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 04:43:18.085005 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079635    2579 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 04:43:18.085005 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079638    2579 flags.go:64] FLAG: --help="false"
Oct 30 04:43:18.085005 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079641    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-135-98.ec2.internal"
Oct 30 04:43:18.085005 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079644    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 04:43:18.085005 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079647    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 04:43:18.085005 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079650    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 04:43:18.085005 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079654    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 04:43:18.085005 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079657    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 04:43:18.085578 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079660    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 04:43:18.085578 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079663    2579 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 04:43:18.085578 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079666    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 04:43:18.085578 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079669    2579 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 04:43:18.085578 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079672    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 04:43:18.085578 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079675    2579 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 04:43:18.085578 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079678    2579 flags.go:64] FLAG: --kube-reserved=""
Oct 30 04:43:18.085578 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079681    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 04:43:18.085578 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079684    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 04:43:18.085578 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079689    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 04:43:18.085578 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079692    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 04:43:18.085578 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079695    2579 flags.go:64] FLAG: --lock-file=""
Oct 30 04:43:18.085578 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079698    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 04:43:18.085578 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079701    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 04:43:18.085578 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079704    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 04:43:18.085578 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079710    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 04:43:18.085578 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079713    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 04:43:18.085578 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079716    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 04:43:18.085578 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079719    2579 flags.go:64] FLAG: --logging-format="text"
Oct 30 04:43:18.085578 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079722    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 04:43:18.085578 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079726    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 04:43:18.085578 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079729    2579 flags.go:64] FLAG: --manifest-url=""
Oct 30 04:43:18.085578 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079732    2579 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 04:43:18.085578 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079737    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 04:43:18.085578 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079740    2579 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 04:43:18.086189 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079755    2579 flags.go:64] FLAG: --max-pods="110"
Oct 30 04:43:18.086189 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079759    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 04:43:18.086189 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079762    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 04:43:18.086189 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079765    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 04:43:18.086189 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079768    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 04:43:18.086189 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079772    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 04:43:18.086189 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079774    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 04:43:18.086189 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079777    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 04:43:18.086189 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079785    2579 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 04:43:18.086189 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079788    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 04:43:18.086189 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079791    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 04:43:18.086189 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079794    2579 flags.go:64] FLAG: --pod-cidr=""
Oct 30 04:43:18.086189 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079797    2579 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 04:43:18.086189 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079800    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 04:43:18.086189 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079803    2579 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 04:43:18.086189 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079806    2579 flags.go:64] FLAG: --port="10250"
Oct 30 04:43:18.086189 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079811    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 04:43:18.086189 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079814    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0207b1bc66b173d2c"
Oct 30 04:43:18.086189 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079817    2579 flags.go:64] FLAG: --qos-reserved=""
Oct 30 04:43:18.086189 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079820    2579 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 04:43:18.086189 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079823    2579 flags.go:64] FLAG: --register-node="true"
Oct 30 04:43:18.086189 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079827    2579 flags.go:64] FLAG: --register-with-taints=""
Oct 30 04:43:18.086189 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079831    2579 flags.go:64] FLAG: --registry-burst="10"
Oct 30 04:43:18.086189 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079833    2579 flags.go:64] FLAG: --registry-qps="5"
Oct 30 04:43:18.086189 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079836    2579 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 04:43:18.086189 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079839    2579 flags.go:64] FLAG: --reserved-memory=""
Oct 30 04:43:18.086985 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079843    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 04:43:18.086985 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079847    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 04:43:18.086985 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079850    2579 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 04:43:18.086985 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079854    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 04:43:18.086985 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079857    2579 flags.go:64] FLAG: --runonce="false"
Oct 30 04:43:18.086985 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079862    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 04:43:18.086985 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079865    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 04:43:18.086985 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079868    2579 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 04:43:18.086985 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079871    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 04:43:18.086985 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079874    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 04:43:18.086985 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079877    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 04:43:18.086985 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079881    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 04:43:18.086985 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079884    2579 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 04:43:18.086985 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079887    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 04:43:18.086985 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079889    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 04:43:18.086985 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079892    2579 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 04:43:18.086985 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079895    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 04:43:18.086985 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079898    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 04:43:18.086985 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079901    2579 flags.go:64] FLAG: --system-cgroups=""
Oct 30 04:43:18.086985 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079904    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 04:43:18.086985 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079910    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 04:43:18.086985 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079913    2579 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 04:43:18.086985 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079917    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 04:43:18.086985 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079921    2579 flags.go:64] FLAG: --tls-min-version=""
Oct 30 04:43:18.086985 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079924    2579 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 04:43:18.087893 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079927    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 04:43:18.087893 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079930    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 04:43:18.087893 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079933    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 04:43:18.087893 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079936    2579 flags.go:64] FLAG: --v="2"
Oct 30 04:43:18.087893 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079941    2579 flags.go:64] FLAG: --version="false"
Oct 30 04:43:18.087893 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079945    2579 flags.go:64] FLAG: --vmodule=""
Oct 30 04:43:18.087893 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079949    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 04:43:18.087893 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.079953    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 04:43:18.087893 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080113    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 04:43:18.087893 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080119    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 04:43:18.087893 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080122    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 04:43:18.087893 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080126    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 04:43:18.087893 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080132    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 04:43:18.087893 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080135    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 04:43:18.087893 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080138    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 04:43:18.087893 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080141    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 04:43:18.087893 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080144    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 04:43:18.087893 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080148    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 04:43:18.087893 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080152    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 04:43:18.087893 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080155    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 04:43:18.087893 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080158    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 04:43:18.088533 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080160    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 04:43:18.088533 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080163    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 04:43:18.088533 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080166    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 04:43:18.088533 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080169    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 04:43:18.088533 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080171    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 04:43:18.088533 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080174    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 04:43:18.088533 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080176    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 04:43:18.088533 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080179    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 04:43:18.088533 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080183    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 04:43:18.088533 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080185    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 04:43:18.088533 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080188    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 04:43:18.088533 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080191    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 04:43:18.088533 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080194    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 04:43:18.088533 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080197    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 04:43:18.088533 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080199    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 04:43:18.088533 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080202    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 04:43:18.088533 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080205    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 04:43:18.088533 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080207    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 04:43:18.088533 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080210    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 04:43:18.089082 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080213    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 04:43:18.089082 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080215    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 04:43:18.089082 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080218    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 04:43:18.089082 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080221    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 04:43:18.089082 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080225    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 04:43:18.089082 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080228    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 04:43:18.089082 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080230    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 04:43:18.089082 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080233    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 04:43:18.089082 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080235    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 04:43:18.089082 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080238    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 04:43:18.089082 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080240    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 04:43:18.089082 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080243    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 04:43:18.089082 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080245    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 04:43:18.089082 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080247    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 04:43:18.089082 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080250    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 04:43:18.089082 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080253    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 04:43:18.089082 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080255    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 04:43:18.089082 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080257    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 04:43:18.089082 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080260    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 04:43:18.089579 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080263    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 04:43:18.089579 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080265    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 04:43:18.089579 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080270    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 04:43:18.089579 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080273    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 04:43:18.089579 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080275    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 04:43:18.089579 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080278    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 04:43:18.089579 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080281    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 04:43:18.089579 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080283    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 04:43:18.089579 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080286    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 04:43:18.089579 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080288    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 04:43:18.089579 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080291    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 04:43:18.089579 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080294    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 04:43:18.089579 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080296    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 04:43:18.089579 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080299    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 04:43:18.089579 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080301    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 04:43:18.089579 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080303    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 04:43:18.089579 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080307    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 04:43:18.089579 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080311    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 04:43:18.089579 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080313    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 04:43:18.089579 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080315    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 04:43:18.090074 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080318    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 04:43:18.090074 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080320    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 04:43:18.090074 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080323    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 04:43:18.090074 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080325    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 04:43:18.090074 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080328    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 04:43:18.090074 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080330    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 04:43:18.090074 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080333    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 04:43:18.090074 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080336    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 04:43:18.090074 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080339    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 04:43:18.090074 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080341    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 04:43:18.090074 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080344    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 04:43:18.090074 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080346    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 04:43:18.090074 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080348    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 04:43:18.090074 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080351    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 04:43:18.090074 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080355    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 04:43:18.090074 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080357    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 04:43:18.090074 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080360    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 04:43:18.090074 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080363    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 04:43:18.090074 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080365    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 04:43:18.090074 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080368    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 04:43:18.090554 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080370    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 04:43:18.090554 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080373    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 04:43:18.090554 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080375    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 04:43:18.090554 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080378    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 04:43:18.090554 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080381    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 04:43:18.090554 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080383    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 04:43:18.090554 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080385    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 04:43:18.090554 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080388    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 04:43:18.090554 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080391    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 04:43:18.090554 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080395    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 04:43:18.090554 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080398    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 04:43:18.090554 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080400    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 04:43:18.090554 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080403    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 04:43:18.090554 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080405    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 04:43:18.090554 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080408    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 04:43:18.090554 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.080434    2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 04:43:18.090947 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080471    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 04:43:18.090947 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080476    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 04:43:18.090947 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080479    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 04:43:18.090947 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080483    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 04:43:18.090947 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080486    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 04:43:18.090947 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080488    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 04:43:18.090947 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080491    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 04:43:18.090947 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080494    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 04:43:18.090947 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080496    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 04:43:18.090947 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080499    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 04:43:18.090947 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080502    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 04:43:18.090947 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080506    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 04:43:18.090947 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080508    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 04:43:18.090947 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080511    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 04:43:18.090947 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080513    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 04:43:18.090947 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080516    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 04:43:18.090947 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080519    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 04:43:18.090947 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080522    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 04:43:18.090947 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080524    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 04:43:18.091393 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080527    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 04:43:18.091393 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080530    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 04:43:18.091393 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080532    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 04:43:18.091393 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080535    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 04:43:18.091393 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080537    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 04:43:18.091393 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080540    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 04:43:18.091393 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080543    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 04:43:18.091393 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080546    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 04:43:18.091393 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080549    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 04:43:18.091393 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080552    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 04:43:18.091393 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080554    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 04:43:18.091393 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080557    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 04:43:18.091393 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080560    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 04:43:18.091393 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080562    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 04:43:18.091393 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080565    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 04:43:18.091393 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080567    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 04:43:18.091393 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080570    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 04:43:18.091393 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080572    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 04:43:18.091393 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080575    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 04:43:18.091847 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080578    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 04:43:18.091847 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080580    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 04:43:18.091847 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080582    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 04:43:18.091847 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080585    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 04:43:18.091847 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080587    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 04:43:18.091847 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080591    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 04:43:18.091847 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080594    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 04:43:18.091847 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080596    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 04:43:18.091847 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080599    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 04:43:18.091847 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080601    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 04:43:18.091847 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080604    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 04:43:18.091847 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080606    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 04:43:18.091847 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080609    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 04:43:18.091847 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080611    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 04:43:18.091847 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080614    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 04:43:18.091847 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080618    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 04:43:18.091847 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080620    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 04:43:18.091847 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080623    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 04:43:18.091847 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080625    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 04:43:18.091847 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080629    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 04:43:18.092318 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080632    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 04:43:18.092318 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080635    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 04:43:18.092318 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080637    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 04:43:18.092318 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080640    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 04:43:18.092318 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080642    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 04:43:18.092318 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080645    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 04:43:18.092318 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080647    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 04:43:18.092318 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080650    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 04:43:18.092318 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080652    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 04:43:18.092318 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080655    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 04:43:18.092318 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080657    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 04:43:18.092318 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080660    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 04:43:18.092318 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080662    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 04:43:18.092318 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080665    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 04:43:18.092318 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080667    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 04:43:18.092318 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080670    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 04:43:18.092318 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080672    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 04:43:18.092318 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080677    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 04:43:18.092318 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080679    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 04:43:18.092860 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080682    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 04:43:18.092860 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080685    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 04:43:18.092860 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080687    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 04:43:18.092860 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080691    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 04:43:18.092860 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080693    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 04:43:18.092860 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080696    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 04:43:18.092860 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080699    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 04:43:18.092860 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080702    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 04:43:18.092860 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080706    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 04:43:18.092860 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080708    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 04:43:18.092860 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080711    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 04:43:18.092860 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080713    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 04:43:18.092860 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080717    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 04:43:18.092860 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080722    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 04:43:18.092860 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080727    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 04:43:18.092860 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080730    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 04:43:18.092860 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080733    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 04:43:18.092860 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080736    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 04:43:18.092860 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080739    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 04:43:18.093304 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080741    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 04:43:18.093304 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080755    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 04:43:18.093304 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080758    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 04:43:18.093304 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080760    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 04:43:18.093304 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080763    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 04:43:18.093304 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080766    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 04:43:18.093304 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080768    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 04:43:18.093304 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080771    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 04:43:18.093304 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080773    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 04:43:18.093304 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.080776    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 04:43:18.093304 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.089116    2579 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 04:43:18.093304 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.089132    2579 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 04:43:18.093304 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.089873    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 04:43:18.093304 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.089881    2579 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 04:43:18.093304 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.089940    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 04:43:18.093304 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.089946    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 04:43:18.093304 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.089950    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 04:43:18.093304 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.089953    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 04:43:18.093304 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.089956    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 04:43:18.093304 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.089958    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 04:43:18.093818 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.089961    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 04:43:18.093818 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.089964    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 04:43:18.093818 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.089967    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 04:43:18.093818 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.089969    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 04:43:18.093818 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.089972    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 04:43:18.093818 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.089975    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 04:43:18.093818 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.089978    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 04:43:18.093818 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.089980    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 04:43:18.093818 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.089983    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 04:43:18.093818 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.089986    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 04:43:18.093818 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.089989    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 04:43:18.093818 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.089991    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 04:43:18.093818 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.089994    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 04:43:18.093818 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.089997    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 04:43:18.093818 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.089999    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 04:43:18.093818 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090002    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 04:43:18.093818 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090004    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 04:43:18.093818 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090007    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 04:43:18.093818 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090010    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 04:43:18.094284 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090013    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 04:43:18.094284 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090016    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 04:43:18.094284 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090020    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 04:43:18.094284 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090023    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 04:43:18.094284 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090026    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 04:43:18.094284 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090028    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 04:43:18.094284 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090031    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 04:43:18.094284 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090034    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 04:43:18.094284 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090036    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 04:43:18.094284 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090039    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 04:43:18.094284 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090042    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 04:43:18.094284 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090044    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 04:43:18.094284 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090047    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 04:43:18.094284 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090049    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 04:43:18.094284 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090052    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 04:43:18.094284 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090055    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 04:43:18.094284 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090057    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 04:43:18.094284 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090060    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 04:43:18.094284 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090063    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 04:43:18.094811 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090065    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 04:43:18.094811 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090068    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 04:43:18.094811 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090070    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 04:43:18.094811 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090073    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 04:43:18.094811 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090076    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 04:43:18.094811 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090079    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 04:43:18.094811 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090082    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 04:43:18.094811 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090084    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 04:43:18.094811 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090087    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 04:43:18.094811 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090090    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 04:43:18.094811 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090093    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 04:43:18.094811 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090096    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 04:43:18.094811 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090098    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 04:43:18.094811 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090101    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 04:43:18.094811 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090104    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 04:43:18.094811 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090107    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 04:43:18.094811 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090110    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 04:43:18.094811 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090113    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 04:43:18.094811 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090116    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 04:43:18.094811 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090118    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 04:43:18.095288 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090121    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 04:43:18.095288 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090124    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 04:43:18.095288 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090126    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 04:43:18.095288 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090129    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 04:43:18.095288 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090131    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 04:43:18.095288 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090134    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 04:43:18.095288 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090137    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 04:43:18.095288 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090139    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 04:43:18.095288 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090142    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 04:43:18.095288 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090145    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 04:43:18.095288 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090147    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 04:43:18.095288 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090150    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 04:43:18.095288 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090153    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 04:43:18.095288 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090155    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 04:43:18.095288 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090158    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 04:43:18.095288 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090161    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 04:43:18.095288 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090163    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 04:43:18.095288 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090166    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 04:43:18.095288 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090168    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 04:43:18.095288 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090171    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 04:43:18.095773 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090173    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 04:43:18.095773 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090176    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 04:43:18.095773 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090178    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 04:43:18.095773 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090182    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 04:43:18.095773 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090184    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 04:43:18.095773 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090187    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 04:43:18.095773 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090189    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 04:43:18.095773 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090194    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 04:43:18.095773 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090199    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 04:43:18.095773 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090202    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 04:43:18.095773 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090205    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 04:43:18.095773 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090208    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 04:43:18.095773 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090210    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 04:43:18.095773 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090213    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 04:43:18.095773 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090215    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 04:43:18.095773 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090218    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 04:43:18.095773 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090220    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 04:43:18.095773 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090224    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 04:43:18.095773 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090228    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 04:43:18.096222 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090231    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 04:43:18.096222 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090234    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 04:43:18.096222 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090237    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 04:43:18.096222 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.090263    2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 04:43:18.096222 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090424    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 04:43:18.096222 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090432    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 04:43:18.096222 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090435    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 04:43:18.096222 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090438    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 04:43:18.096222 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090441    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 04:43:18.096222 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090448    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 04:43:18.096222 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090452    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 04:43:18.096222 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090455    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 04:43:18.096222 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090457    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 04:43:18.096222 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090460    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 04:43:18.096222 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090462    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 04:43:18.096222 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090465    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 04:43:18.096598 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090468    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 04:43:18.096598 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090471    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 04:43:18.096598 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090474    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 04:43:18.096598 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090476    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 04:43:18.096598 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090479    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 04:43:18.096598 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090482    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 04:43:18.096598 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090485    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 04:43:18.096598 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090488    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 04:43:18.096598 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090491    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 04:43:18.096598 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090493    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 04:43:18.096598 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090496    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 04:43:18.096598 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090498    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 04:43:18.096598 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090501    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 04:43:18.096598 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090503    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 04:43:18.096598 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090506    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 04:43:18.096598 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090508    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 04:43:18.096598 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090511    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 04:43:18.096598 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090514    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 04:43:18.096598 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090516    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 04:43:18.097093 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090519    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 04:43:18.097093 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090521    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 04:43:18.097093 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090524    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 04:43:18.097093 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090527    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 04:43:18.097093 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090531    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 04:43:18.097093 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090534    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 04:43:18.097093 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090537    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 04:43:18.097093 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090540    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 04:43:18.097093 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090543    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 04:43:18.097093 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090546    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 04:43:18.097093 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090548    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 04:43:18.097093 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090551    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 04:43:18.097093 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090553    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 04:43:18.097093 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090556    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 04:43:18.097093 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090559    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 04:43:18.097093 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090562    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 04:43:18.097093 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090565    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 04:43:18.097093 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090568    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 04:43:18.097093 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090570    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 04:43:18.097536 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090573    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 04:43:18.097536 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090575    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 04:43:18.097536 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090578    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 04:43:18.097536 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090580    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 04:43:18.097536 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090583    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 04:43:18.097536 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090585    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 04:43:18.097536 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090588    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 04:43:18.097536 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090590    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 04:43:18.097536 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090593    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 04:43:18.097536 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090595    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 04:43:18.097536 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090598    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 04:43:18.097536 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090600    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 04:43:18.097536 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090603    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 04:43:18.097536 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090605    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 04:43:18.097536 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090608    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 04:43:18.097536 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090611    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 04:43:18.097536 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090613    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 04:43:18.097536 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090616    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 04:43:18.097536 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090620    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 04:43:18.098043 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090623    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 04:43:18.098043 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090626    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 04:43:18.098043 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090630    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 04:43:18.098043 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090633    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 04:43:18.098043 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090635    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 04:43:18.098043 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090638    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 04:43:18.098043 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090640    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 04:43:18.098043 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090643    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 04:43:18.098043 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090646    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 04:43:18.098043 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090648    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 04:43:18.098043 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090651    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 04:43:18.098043 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090653    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 04:43:18.098043 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090656    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 04:43:18.098043 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090659    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 04:43:18.098043 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090661    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 04:43:18.098043 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090664    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 04:43:18.098043 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090666    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 04:43:18.098043 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090669    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 04:43:18.098043 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090671    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 04:43:18.098043 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090674    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 04:43:18.098644 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090677    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 04:43:18.098644 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090680    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 04:43:18.098644 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090683    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 04:43:18.098644 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090685    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 04:43:18.098644 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090688    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 04:43:18.098644 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090690    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 04:43:18.098644 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090693    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 04:43:18.098644 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090695    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 04:43:18.098644 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090698    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 04:43:18.098644 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090700    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 04:43:18.098644 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090702    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 04:43:18.098644 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090705    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 04:43:18.098644 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090707    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 04:43:18.098644 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090710    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 04:43:18.098644 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090712    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 04:43:18.098644 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090715    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 04:43:18.098644 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090718    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 04:43:18.099072 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.090740    2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 04:43:18.099072 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090785    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 04:43:18.099072 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090790    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 04:43:18.099072 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090793    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 04:43:18.099072 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090797    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 04:43:18.099072 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090800    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 04:43:18.099072 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090803    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 04:43:18.099072 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090806    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 04:43:18.099072 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090808    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 04:43:18.099072 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090811    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 04:43:18.099072 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090814    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 04:43:18.099072 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090816    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 04:43:18.099072 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090819    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 04:43:18.099072 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090821    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 04:43:18.099072 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090823    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 04:43:18.099072 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090826    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 04:43:18.099443 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090829    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 04:43:18.099443 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090831    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 04:43:18.099443 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090833    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 04:43:18.099443 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090836    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 04:43:18.099443 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090838    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 04:43:18.099443 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090841    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 04:43:18.099443 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090844    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 04:43:18.099443 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090846    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 04:43:18.099443 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090849    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 04:43:18.099443 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090851    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 04:43:18.099443 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090854    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 04:43:18.099443 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090857    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 04:43:18.099443 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090859    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 04:43:18.099443 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090862    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 04:43:18.099443 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090865    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 04:43:18.099443 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090868    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 04:43:18.099443 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090871    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 04:43:18.099443 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090873    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 04:43:18.099443 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090876    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 04:43:18.099443 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090878    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 04:43:18.099940 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090881    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 04:43:18.099940 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090884    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 04:43:18.099940 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090886    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 04:43:18.099940 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090889    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 04:43:18.099940 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090891    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 04:43:18.099940 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090893    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 04:43:18.099940 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090896    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 04:43:18.099940 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090898    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 04:43:18.099940 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090901    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 04:43:18.099940 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090903    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 04:43:18.099940 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090906    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 04:43:18.099940 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090908    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 04:43:18.099940 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090911    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 04:43:18.099940 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090913    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 04:43:18.099940 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090916    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 04:43:18.099940 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090918    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 04:43:18.099940 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090922    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 04:43:18.099940 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090925    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 04:43:18.099940 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090928    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 04:43:18.100373 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090931    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 04:43:18.100373 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090934    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 04:43:18.100373 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090937    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 04:43:18.100373 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090941    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 04:43:18.100373 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090943    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 04:43:18.100373 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090946    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 04:43:18.100373 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090952    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 04:43:18.100373 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090956    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 04:43:18.100373 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090959    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 04:43:18.100373 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090963    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 04:43:18.100373 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090966    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 04:43:18.100373 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090968    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 04:43:18.100373 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090971    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 04:43:18.100373 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090973    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 04:43:18.100373 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090976    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 04:43:18.100373 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090979    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 04:43:18.100373 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090983    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 04:43:18.100373 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090985    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 04:43:18.100373 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090988    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 04:43:18.100373 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090990    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 04:43:18.100963 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090993    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 04:43:18.100963 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090995    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 04:43:18.100963 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.090998    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 04:43:18.100963 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.091000    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 04:43:18.100963 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.091003    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 04:43:18.100963 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.091005    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 04:43:18.100963 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.091008    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 04:43:18.100963 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.091011    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 04:43:18.100963 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.091014    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 04:43:18.100963 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.091016    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 04:43:18.100963 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.091020    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 04:43:18.100963 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.091023    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 04:43:18.100963 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.091026    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 04:43:18.100963 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.091028    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 04:43:18.100963 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.091031    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 04:43:18.100963 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.091033    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 04:43:18.100963 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.091036    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 04:43:18.100963 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.091038    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 04:43:18.100963 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.091042    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 04:43:18.101490 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.091044    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 04:43:18.101490 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.091047    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 04:43:18.101490 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.091049    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 04:43:18.101490 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.091052    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 04:43:18.101490 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.091055    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 04:43:18.101490 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.091057    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 04:43:18.101490 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.091060    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 04:43:18.101490 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.091062    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 04:43:18.101490 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.091064    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 04:43:18.101490 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.091068    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 04:43:18.101490 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.091071    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 04:43:18.101490 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.091073    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 04:43:18.101490 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:18.091077    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 04:43:18.101490 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.091125    2579 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 04:43:18.101490 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.093637    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 04:43:18.101490 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.094665    2579 server.go:1014] "Starting client certificate rotation"
Oct 30 04:43:18.101490 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.094767    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 04:43:18.101490 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.094804    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 04:43:18.101490 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.096863    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 04:43:18.101994 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.097965    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 04:43:18.110548 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.110531    2579 log.go:25] "Validated CRI v1 runtime API"
Oct 30 04:43:18.113529 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.113516    2579 log.go:25] "Validated CRI v1 image API"
Oct 30 04:43:18.114337 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.114304    2579 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 04:43:18.117228 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.117205    2579 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 b2f2c13c-3afe-4217-bacb-b7de744e0a49:/dev/nvme0n1p3 ecc471dc-8048-4b0a-82b0-9f40149b53b3:/dev/nvme0n1p4]
Oct 30 04:43:18.117278 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.117231    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 04:43:18.124487 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.124363    2579 manager.go:217] Machine: {Timestamp:2026-10-30 04:43:18.12258813 +0000 UTC m=+0.375707466 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3119585 MemoryCapacity:33164324864 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21f93a24d2f5567a3b9cb0461d9fd6 SystemUUID:ec21f93a-24d2-f556-7a3b-9cb0461d9fd6 BootID:7b07d1f0-e52b-4d8c-a7f1-03975f01a029 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582164480 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582160384 Type:vfs Inodes:4048379 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:dc:18:ca:55 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:dc:18:ca:55 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c2:12:8d:11:55:db Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164324864 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 04:43:18.125110 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.125098    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 04:43:18.125228 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.125210    2579 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 04:43:18.128756 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.128712    2579 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 04:43:18.128874 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.128739    2579 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-98.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 04:43:18.128922 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.128876    2579 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 04:43:18.128922 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.128884    2579 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 04:43:18.128922 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.128916    2579 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 04:43:18.129010 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.128933    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 04:43:18.130223 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.130213    2579 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 04:43:18.134901 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.134889    2579 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 04:43:18.135016 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.135007    2579 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 04:43:18.135898 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.135886    2579 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 04:43:18.135935 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.135905    2579 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 04:43:18.135935 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.135919    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 04:43:18.135935 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.135931    2579 kubelet.go:405] "Adding apiserver pod source"
Oct 30 04:43:18.136028 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.135941    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 04:43:18.136395 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.136302    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 04:43:18.138782 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.138764    2579 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 04:43:18.145520 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.145507    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 04:43:18.146323 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.146311    2579 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 04:43:18.146360 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.146342    2579 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 04:43:18.146497 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.146485    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 04:43:18.146526 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.146507    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 04:43:18.146526 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.146517    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 04:43:18.146623 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.146524    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 04:43:18.146623 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.146534    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 04:43:18.146623 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.146546    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 04:43:18.147228 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.147218    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 04:43:18.147256 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.147229    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 04:43:18.147256 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.147236    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 04:43:18.147256 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.147243    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 04:43:18.147256 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.147252    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 04:43:18.147374 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.147261    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 04:43:18.148009 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.148001    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 04:43:18.148043 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.148011    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 04:43:18.151780 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.151768    2579 server.go:1263] "Started kubelet"
Oct 30 04:43:18.151933 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.151909    2579 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 04:43:18.152137 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.151883    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 04:43:18.152194 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.152161    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 04:43:18.152431 ip-10-0-135-98 systemd[1]: Started Kubernetes Kubelet.
Oct 30 04:43:18.153592 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.153571    2579 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 04:43:18.154850 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.154835    2579 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 04:43:18.158775 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:18.158742    2579 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 04:43:18.160541 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.160524    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 04:43:18.161128 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.161110    2579 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 04:43:18.161334 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:18.161290    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-98.ec2.internal\" not found"
Oct 30 04:43:18.161832 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.161817    2579 factory.go:55] Registering systemd factory
Oct 30 04:43:18.161832 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.161831    2579 factory.go:223] Registration of the systemd container factory successfully
Oct 30 04:43:18.162034 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.162020    2579 factory.go:153] Registering CRI-O factory
Oct 30 04:43:18.162085 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.162037    2579 factory.go:223] Registration of the crio container factory successfully
Oct 30 04:43:18.162085 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.162051    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 04:43:18.162085 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.162053    2579 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 04:43:18.162085 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.162069    2579 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 04:43:18.162085 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.162084    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 04:43:18.162313 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.162097    2579 factory.go:103] Registering Raw factory
Oct 30 04:43:18.162313 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.162107    2579 manager.go:1199] Started watching for new ooms in manager
Oct 30 04:43:18.162313 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.162176    2579 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 04:43:18.162313 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.162184    2579 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 04:43:18.163300 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.163287    2579 manager.go:319] Starting recovery of all containers
Oct 30 04:43:18.172804 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.172779    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-98.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 04:43:18.173990 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:18.172780    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-98.ec2.internal.18e3350b09f79bfb  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-98.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-98.ec2.internal,},FirstTimestamp:2026-10-30 04:43:18.151724027 +0000 UTC m=+0.404843363,LastTimestamp:2026-10-30 04:43:18.151724027 +0000 UTC m=+0.404843363,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-98.ec2.internal,}"
Oct 30 04:43:18.174364 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.174349    2579 manager.go:324] Recovery completed
Oct 30 04:43:18.178915 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.178903    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 04:43:18.180898 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.180874    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-98.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 04:43:18.180956 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.180910    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-98.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 04:43:18.180956 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.180921    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-98.ec2.internal" event="NodeHasSufficientPID"
Oct 30 04:43:18.182214 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.182199    2579 cpu_manager.go:226] "Starting" policy="none"
Oct 30 04:43:18.182214 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.182210    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 04:43:18.182340 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.182213    2579 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 04:43:18.182340 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.182270    2579 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 04:43:18.182887 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.182874    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-98.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 04:43:18.182932 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.182900    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-98.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 04:43:18.182932 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.182913    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-98.ec2.internal" event="NodeHasSufficientPID"
Oct 30 04:43:18.184530 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.184517    2579 policy_none.go:50] "Start"
Oct 30 04:43:18.184571 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.184533    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 04:43:18.184571 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.184543    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 04:43:18.186485 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.186464    2579 policy_none.go:44] "Start"
Oct 30 04:43:18.186556 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:18.186486    2579 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-98.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 04:43:18.189941 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.189923    2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 04:43:18.190022 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.189962    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2nrh7"
Oct 30 04:43:18.202347 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:18.202318    2579 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 04:43:18.202547 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:18.202507    2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-98.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 04:43:18.202907 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:18.202336    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-98.ec2.internal.18e3350b0bb4c1b3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-98.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-98.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-98.ec2.internal,},FirstTimestamp:2026-10-30 04:43:18.180897203 +0000 UTC m=+0.434016539,LastTimestamp:2026-10-30 04:43:18.180897203 +0000 UTC m=+0.434016539,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-98.ec2.internal,}"
Oct 30 04:43:18.203050 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:18.203024    2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 04:43:18.213454 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.213424    2579 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 04:43:18.239666 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.217219    2579 manager.go:341] "Starting Device Plugin manager"
Oct 30 04:43:18.239666 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:18.217245    2579 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 04:43:18.239666 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.217253    2579 server.go:87] "Starting device plugin registration server"
Oct 30 04:43:18.239666 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.217433    2579 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 04:43:18.239666 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.217442    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 04:43:18.239666 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.217570    2579 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 04:43:18.239666 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.217640    2579 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 04:43:18.239666 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.217647    2579 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 04:43:18.239666 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:18.218236    2579 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 04:43:18.239666 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:18.218281    2579 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-98.ec2.internal\" not found"
Oct 30 04:43:18.239666 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.229602    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 04:43:18.239666 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.230797    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 04:43:18.239666 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.230822    2579 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 04:43:18.239666 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.230842    2579 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 04:43:18.239666 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:18.230871    2579 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 04:43:18.245297 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.245282    2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 04:43:18.317633 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.317614    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 04:43:18.318439 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.318424    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-98.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 04:43:18.318529 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.318450    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-98.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 04:43:18.318529 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.318461    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-98.ec2.internal" event="NodeHasSufficientPID"
Oct 30 04:43:18.318529 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.318483    2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-98.ec2.internal"
Oct 30 04:43:18.331379 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.331321    2579 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-135-98.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-98.ec2.internal"]
Oct 30 04:43:18.331379 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.331375    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 04:43:18.332095 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.332072    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-98.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 04:43:18.332186 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.332102    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-98.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 04:43:18.332186 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.332112    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-98.ec2.internal" event="NodeHasSufficientPID"
Oct 30 04:43:18.334376 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.334358    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 04:43:18.334500 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.334487    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-98.ec2.internal"
Oct 30 04:43:18.334537 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.334512    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 04:43:18.335051 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.335033    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-98.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 04:43:18.335145 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.335061    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-98.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 04:43:18.335145 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.335070    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-98.ec2.internal" event="NodeHasSufficientPID"
Oct 30 04:43:18.335145 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.335033    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-98.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 04:43:18.335296 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.335148    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-98.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 04:43:18.335296 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.335169    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-98.ec2.internal" event="NodeHasSufficientPID"
Oct 30 04:43:18.336651 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.336630    2579 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-98.ec2.internal"
Oct 30 04:43:18.336714 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:18.336650    2579 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-98.ec2.internal\": node \"ip-10-0-135-98.ec2.internal\" not found"
Oct 30 04:43:18.337677 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.337659    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-98.ec2.internal"
Oct 30 04:43:18.337774 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.337694    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 04:43:18.338450 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.338428    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-98.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 04:43:18.338553 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.338457    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-98.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 04:43:18.338553 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.338466    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-98.ec2.internal" event="NodeHasSufficientPID"
Oct 30 04:43:18.360383 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:18.360359    2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-98.ec2.internal\" not found" node="ip-10-0-135-98.ec2.internal"
Oct 30 04:43:18.363936 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.363917    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/38fe9c19c3aaab70dd96caa47ec7e827-config\") pod \"kube-apiserver-proxy-ip-10-0-135-98.ec2.internal\" (UID: \"38fe9c19c3aaab70dd96caa47ec7e827\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-98.ec2.internal"
Oct 30 04:43:18.364005 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.363941    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b4286e30fd1fc24b304f69bb80b9a852-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-98.ec2.internal\" (UID: \"b4286e30fd1fc24b304f69bb80b9a852\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-98.ec2.internal"
Oct 30 04:43:18.364005 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.363956    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b4286e30fd1fc24b304f69bb80b9a852-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-98.ec2.internal\" (UID: \"b4286e30fd1fc24b304f69bb80b9a852\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-98.ec2.internal"
Oct 30 04:43:18.366791 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:18.366775    2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-98.ec2.internal\" not found" node="ip-10-0-135-98.ec2.internal"
Oct 30 04:43:18.386304 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:18.386285    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-98.ec2.internal\" not found"
Oct 30 04:43:18.464855 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.464831    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b4286e30fd1fc24b304f69bb80b9a852-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-98.ec2.internal\" (UID: \"b4286e30fd1fc24b304f69bb80b9a852\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-98.ec2.internal"
Oct 30 04:43:18.464975 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.464886    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b4286e30fd1fc24b304f69bb80b9a852-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-98.ec2.internal\" (UID: \"b4286e30fd1fc24b304f69bb80b9a852\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-98.ec2.internal"
Oct 30 04:43:18.464975 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.464855    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b4286e30fd1fc24b304f69bb80b9a852-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-98.ec2.internal\" (UID: \"b4286e30fd1fc24b304f69bb80b9a852\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-98.ec2.internal"
Oct 30 04:43:18.464975 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.464917    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b4286e30fd1fc24b304f69bb80b9a852-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-98.ec2.internal\" (UID: \"b4286e30fd1fc24b304f69bb80b9a852\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-98.ec2.internal"
Oct 30 04:43:18.464975 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.464943    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/38fe9c19c3aaab70dd96caa47ec7e827-config\") pod \"kube-apiserver-proxy-ip-10-0-135-98.ec2.internal\" (UID: \"38fe9c19c3aaab70dd96caa47ec7e827\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-98.ec2.internal"
Oct 30 04:43:18.464975 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.464969    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/38fe9c19c3aaab70dd96caa47ec7e827-config\") pod \"kube-apiserver-proxy-ip-10-0-135-98.ec2.internal\" (UID: \"38fe9c19c3aaab70dd96caa47ec7e827\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-98.ec2.internal"
Oct 30 04:43:18.487047 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:18.487028    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-98.ec2.internal\" not found"
Oct 30 04:43:18.588047 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:18.587990    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-98.ec2.internal\" not found"
Oct 30 04:43:18.665512 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.665486    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-98.ec2.internal"
Oct 30 04:43:18.671718 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:18.671701    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-98.ec2.internal"
Oct 30 04:43:18.688996 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:18.688844    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-98.ec2.internal\" not found"
Oct 30 04:43:18.789431 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:18.789396    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-98.ec2.internal\" not found"
Oct 30 04:43:18.889974 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:18.889917    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-98.ec2.internal\" not found"
Oct 30 04:43:18.990380 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:18.990351    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-98.ec2.internal\" not found"
Oct 30 04:43:20.367300 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:20.367280    2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 04:43:20.412580 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:20.412560    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-98.ec2.internal\" not found"
Oct 30 04:43:20.415727 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:20.415701    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 04:43:20.415832 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:20.415809    2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 04:43:20.415896 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:20.415811    2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 04:43:20.513459 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:20.513399    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-98.ec2.internal\" not found"
Oct 30 04:43:20.519564 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:20.519533    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 04:38:19 +0000 UTC" deadline="2028-06-20 11:40:45.683859227 +0000 UTC"
Oct 30 04:43:20.519564 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:20.519560    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14382h57m25.164302229s"
Oct 30 04:43:20.576107 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:20.576080    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4286e30fd1fc24b304f69bb80b9a852.slice/crio-9b517736b4c15deb64649ad76e845dfc28dd4505bb717cdb56f37e5f9e23363a WatchSource:0}: Error finding container 9b517736b4c15deb64649ad76e845dfc28dd4505bb717cdb56f37e5f9e23363a: Status 404 returned error can't find the container with id 9b517736b4c15deb64649ad76e845dfc28dd4505bb717cdb56f37e5f9e23363a
Oct 30 04:43:20.576345 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:20.576325    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38fe9c19c3aaab70dd96caa47ec7e827.slice/crio-03cda3c4c09016f9eed112c104471848e90fd8df45e68b3d01027033385a64bf WatchSource:0}: Error finding container 03cda3c4c09016f9eed112c104471848e90fd8df45e68b3d01027033385a64bf: Status 404 returned error can't find the container with id 03cda3c4c09016f9eed112c104471848e90fd8df45e68b3d01027033385a64bf
Oct 30 04:43:20.580690 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:20.580675    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 04:43:20.613549 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:20.613524    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-98.ec2.internal\" not found"
Oct 30 04:43:20.714110 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:20.714069    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-98.ec2.internal\" not found"
Oct 30 04:43:20.717225 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:20.717208    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 04:43:20.718368 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:20.718352    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-98.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 04:43:20.718436 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:20.718380    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-98.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 04:43:20.718436 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:20.718392    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-98.ec2.internal" event="NodeHasSufficientPID"
Oct 30 04:43:20.740378 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:20.740364    2579 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 04:43:20.783464 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:20.783420    2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-98.ec2.internal"
Oct 30 04:43:20.803931 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:20.803917    2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 04:43:20.806023 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:20.806012    2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-98.ec2.internal"
Oct 30 04:43:20.828035 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:20.828019    2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 04:43:21.143955 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.143702    2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 04:43:21.351808 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.351783    2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 04:43:21.458629 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.458563    2579 apiserver.go:52] "Watching apiserver"
Oct 30 04:43:21.483205 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.482590    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 04:43:21.487423 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.487399    2579 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 04:43:21.487800 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.487767    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-dgsdt","openshift-ovn-kubernetes/ovnkube-node-v4bnv","kube-system/global-pull-secret-syncer-tt5mj","kube-system/konnectivity-agent-6bpl4","openshift-cluster-node-tuning-operator/tuned-qtb8v","openshift-multus/multus-c9tpc","openshift-multus/network-metrics-daemon-vjfrn","openshift-network-operator/iptables-alerter-st55p","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhsc7","openshift-multus/multus-additional-cni-plugins-tnwc6","openshift-image-registry/node-ca-zqs6v","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-98.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-98.ec2.internal","openshift-dns/node-resolver-gndxs"]
Oct 30 04:43:21.490737 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.490709    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vjfrn"
Oct 30 04:43:21.490838 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:21.490778    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vjfrn" podUID="7f75d879-5e31-41fd-838c-b1f0408cb45a"
Oct 30 04:43:21.492643 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.492621    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zqs6v"
Oct 30 04:43:21.494883 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.494866    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tt5mj"
Oct 30 04:43:21.494990 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:21.494944    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tt5mj" podUID="41a4ec4b-b407-4316-9238-dee6be432d3c"
Oct 30 04:43:21.497457 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.497120    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6bpl4"
Oct 30 04:43:21.499332 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.499314    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:43:21.501593 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.501572    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-c9tpc"
Oct 30 04:43:21.504183 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.504164    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhsc7"
Oct 30 04:43:21.511477 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.511321    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gndxs"
Oct 30 04:43:21.512026 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.512010    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:21.514111 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.512992    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-45bnw\""
Oct 30 04:43:21.514111 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.513035    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 04:43:21.514262 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.513050    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 04:43:21.514262 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.513071    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 04:43:21.514262 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.513149    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 04:43:21.514262 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.513157    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 04:43:21.514262 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.513206    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-g88vf\""
Oct 30 04:43:21.514492 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.514482    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 04:43:21.514552 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.514514    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rhd5l\""
Oct 30 04:43:21.514751 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.514733    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 04:43:21.515837 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.515820    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tnwc6"
Oct 30 04:43:21.517138 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.517120    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 04:43:21.517244 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.517213    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 04:43:21.517301 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.517265    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 04:43:21.517301 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.517235    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-jzfzk\""
Oct 30 04:43:21.517402 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.517357    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 04:43:21.519418 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.519382    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgsdt"
Oct 30 04:43:21.519505 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:21.519434    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgsdt" podUID="3b1cae8a-12cf-4b59-85f0-a63f324fc8e1"
Oct 30 04:43:21.521150 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.521122    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 04:43:21.521437 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.521417    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-st55p"
Oct 30 04:43:21.556967 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.556921    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-98.ec2.internal" event={"ID":"b4286e30fd1fc24b304f69bb80b9a852","Type":"ContainerStarted","Data":"9b517736b4c15deb64649ad76e845dfc28dd4505bb717cdb56f37e5f9e23363a"}
Oct 30 04:43:21.558419 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.558396    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-98.ec2.internal" event={"ID":"38fe9c19c3aaab70dd96caa47ec7e827","Type":"ContainerStarted","Data":"03cda3c4c09016f9eed112c104471848e90fd8df45e68b3d01027033385a64bf"}
Oct 30 04:43:21.585268 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.585249    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 04:43:21.595450 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.595419    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/672524df-2f52-455b-9423-a58f53c31129-etc-selinux\") pod \"aws-ebs-csi-driver-node-vhsc7\" (UID: \"672524df-2f52-455b-9423-a58f53c31129\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhsc7"
Oct 30 04:43:21.595570 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.595461    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-host-run-multus-certs\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:43:21.595570 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.595492    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-lib-modules\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:43:21.595570 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.595526    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/672524df-2f52-455b-9423-a58f53c31129-sys-fs\") pod \"aws-ebs-csi-driver-node-vhsc7\" (UID: \"672524df-2f52-455b-9423-a58f53c31129\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhsc7"
Oct 30 04:43:21.595735 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.595565    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-systemd-units\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:21.595735 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.595604    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-run-systemd\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:21.595735 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.595638    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f75d879-5e31-41fd-838c-b1f0408cb45a-metrics-certs\") pod \"network-metrics-daemon-vjfrn\" (UID: \"7f75d879-5e31-41fd-838c-b1f0408cb45a\") " pod="openshift-multus/network-metrics-daemon-vjfrn"
Oct 30 04:43:21.595735 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.595663    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/41a4ec4b-b407-4316-9238-dee6be432d3c-kubelet-config\") pod \"global-pull-secret-syncer-tt5mj\" (UID: \"41a4ec4b-b407-4316-9238-dee6be432d3c\") " pod="kube-system/global-pull-secret-syncer-tt5mj"
Oct 30 04:43:21.595735 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.595692    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-host-run-netns\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:43:21.595735 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.595733    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-host-cni-bin\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:21.596033 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.595763    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/885d476c-7568-4b63-ab9d-1da7608706f5-hosts-file\") pod \"node-resolver-gndxs\" (UID: \"885d476c-7568-4b63-ab9d-1da7608706f5\") " pod="openshift-dns/node-resolver-gndxs"
Oct 30 04:43:21.596033 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.595787    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2a3eb4b-a128-4649-bf77-78df2705f147-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tnwc6\" (UID: \"a2a3eb4b-a128-4649-bf77-78df2705f147\") " pod="openshift-multus/multus-additional-cni-plugins-tnwc6"
Oct 30 04:43:21.596033 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.595812    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-sys\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:43:21.596033 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.595858    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-tmp\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:43:21.596033 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.595920    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2a3eb4b-a128-4649-bf77-78df2705f147-system-cni-dir\") pod \"multus-additional-cni-plugins-tnwc6\" (UID: \"a2a3eb4b-a128-4649-bf77-78df2705f147\") " pod="openshift-multus/multus-additional-cni-plugins-tnwc6"
Oct 30 04:43:21.596033 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.595948    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a2a3eb4b-a128-4649-bf77-78df2705f147-cni-binary-copy\") pod \"multus-additional-cni-plugins-tnwc6\" (UID: \"a2a3eb4b-a128-4649-bf77-78df2705f147\") " pod="openshift-multus/multus-additional-cni-plugins-tnwc6"
Oct 30 04:43:21.596033 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.595981    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/73a78a0d-d19e-4a96-b8ce-253d631da703-konnectivity-ca\") pod \"konnectivity-agent-6bpl4\" (UID: \"73a78a0d-d19e-4a96-b8ce-253d631da703\") " pod="kube-system/konnectivity-agent-6bpl4"
Oct 30 04:43:21.596033 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.596003    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-run-ovn\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:21.596033 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.596029    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-host-kubelet\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:21.596453 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.596068    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-etc-kubernetes\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:43:21.596453 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.596093    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-hostroot\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:43:21.596453 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.596124    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-multus-conf-dir\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:43:21.596453 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.596151    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:21.596453 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.596201    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-etc-kubernetes\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:43:21.596453 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.596227    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwcw4\" (UniqueName: \"kubernetes.io/projected/885d476c-7568-4b63-ab9d-1da7608706f5-kube-api-access-kwcw4\") pod \"node-resolver-gndxs\" (UID: \"885d476c-7568-4b63-ab9d-1da7608706f5\") " pod="openshift-dns/node-resolver-gndxs"
Oct 30 04:43:21.596453 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.596252    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxpck\" (UniqueName: \"kubernetes.io/projected/a2a3eb4b-a128-4649-bf77-78df2705f147-kube-api-access-nxpck\") pod \"multus-additional-cni-plugins-tnwc6\" (UID: \"a2a3eb4b-a128-4649-bf77-78df2705f147\") " pod="openshift-multus/multus-additional-cni-plugins-tnwc6"
Oct 30 04:43:21.596453 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.596275    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bf49335f-7b8e-4212-8777-1575cd0929f1-cni-binary-copy\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:43:21.596453 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.596308    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/70169887-cc72-4fb5-96b5-e09e260d7a8b-ovn-node-metrics-cert\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:21.596453 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.596336    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-host\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:43:21.596453 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.596368    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92qd4\" (UniqueName: \"kubernetes.io/projected/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-kube-api-access-92qd4\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:43:21.596453 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.596394    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-host-var-lib-cni-bin\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:43:21.596453 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.596417    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-etc-sysctl-d\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:43:21.596453 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.596449    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29k58\" (UniqueName: \"kubernetes.io/projected/daf6535c-7f33-407a-a5d9-e1c9525c8cb8-kube-api-access-29k58\") pod \"node-ca-zqs6v\" (UID: \"daf6535c-7f33-407a-a5d9-e1c9525c8cb8\") " pod="openshift-image-registry/node-ca-zqs6v"
Oct 30 04:43:21.597126 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.596530    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-node-log\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:21.597126 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.596557    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-etc-modprobe-d\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:43:21.597126 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.596580    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-etc-sysconfig\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:43:21.597126 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.596603    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-host-slash\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:21.597126 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.596628    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/70169887-cc72-4fb5-96b5-e09e260d7a8b-ovnkube-script-lib\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:21.597126 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.596651    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bff48d36-3909-4838-b499-66e5081bcc90-host-slash\") pod \"iptables-alerter-st55p\" (UID: \"bff48d36-3909-4838-b499-66e5081bcc90\") " pod="openshift-network-operator/iptables-alerter-st55p"
Oct 30 04:43:21.597126 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.596676    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49nsv\" (UniqueName: \"kubernetes.io/projected/7f75d879-5e31-41fd-838c-b1f0408cb45a-kube-api-access-49nsv\") pod \"network-metrics-daemon-vjfrn\" (UID: \"7f75d879-5e31-41fd-838c-b1f0408cb45a\") " pod="openshift-multus/network-metrics-daemon-vjfrn"
Oct 30 04:43:21.597126 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.596699    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-etc-tuned\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:43:21.597126 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.596735    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/672524df-2f52-455b-9423-a58f53c31129-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vhsc7\" (UID: \"672524df-2f52-455b-9423-a58f53c31129\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhsc7"
Oct 30 04:43:21.597126 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.596758    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-host-var-lib-kubelet\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:43:21.597126 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.596786    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w8n4\" (UniqueName: \"kubernetes.io/projected/bf49335f-7b8e-4212-8777-1575cd0929f1-kube-api-access-4w8n4\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:43:21.597126 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.596817    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-etc-openvswitch\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:21.597126 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.596842    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bff48d36-3909-4838-b499-66e5081bcc90-iptables-alerter-script\") pod \"iptables-alerter-st55p\" (UID: \"bff48d36-3909-4838-b499-66e5081bcc90\") " pod="openshift-network-operator/iptables-alerter-st55p"
Oct 30 04:43:21.597126 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.596866    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/41a4ec4b-b407-4316-9238-dee6be432d3c-dbus\") pod \"global-pull-secret-syncer-tt5mj\" (UID: \"41a4ec4b-b407-4316-9238-dee6be432d3c\") " pod="kube-system/global-pull-secret-syncer-tt5mj"
Oct 30 04:43:21.597126 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.596891    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-etc-sysctl-conf\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:43:21.597126 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.596934    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-run\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:43:21.597938 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.596957    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/672524df-2f52-455b-9423-a58f53c31129-socket-dir\") pod \"aws-ebs-csi-driver-node-vhsc7\" (UID: \"672524df-2f52-455b-9423-a58f53c31129\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhsc7"
Oct 30 04:43:21.597938 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.596982    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a2a3eb4b-a128-4649-bf77-78df2705f147-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tnwc6\" (UID: \"a2a3eb4b-a128-4649-bf77-78df2705f147\") " pod="openshift-multus/multus-additional-cni-plugins-tnwc6"
Oct 30 04:43:21.597938 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.597007    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a2a3eb4b-a128-4649-bf77-78df2705f147-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tnwc6\" (UID: \"a2a3eb4b-a128-4649-bf77-78df2705f147\") " pod="openshift-multus/multus-additional-cni-plugins-tnwc6"
Oct 30 04:43:21.597938 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.597117    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-cnibin\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:43:21.597938 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.597142    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-var-lib-openvswitch\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:21.597938 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.597166    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-host-cni-netd\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:21.597938 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.597192    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-etc-systemd\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:43:21.597938 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.597216    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/672524df-2f52-455b-9423-a58f53c31129-device-dir\") pod \"aws-ebs-csi-driver-node-vhsc7\" (UID: \"672524df-2f52-455b-9423-a58f53c31129\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhsc7"
Oct 30 04:43:21.597938 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.597245    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfwq8\" (UniqueName: \"kubernetes.io/projected/672524df-2f52-455b-9423-a58f53c31129-kube-api-access-bfwq8\") pod \"aws-ebs-csi-driver-node-vhsc7\" (UID: \"672524df-2f52-455b-9423-a58f53c31129\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhsc7"
Oct 30 04:43:21.597938 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.597270    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/73a78a0d-d19e-4a96-b8ce-253d631da703-agent-certs\") pod \"konnectivity-agent-6bpl4\" (UID: \"73a78a0d-d19e-4a96-b8ce-253d631da703\") " pod="kube-system/konnectivity-agent-6bpl4"
Oct 30 04:43:21.597938 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.597294    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-run-openvswitch\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:21.597938 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.597317    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-log-socket\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:21.597938 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.597340    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/70169887-cc72-4fb5-96b5-e09e260d7a8b-ovnkube-config\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:21.597938 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.597365    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bf49335f-7b8e-4212-8777-1575cd0929f1-multus-daemon-config\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:43:21.597938 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.597399    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-var-lib-kubelet\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:43:21.597938 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.597424    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/daf6535c-7f33-407a-a5d9-e1c9525c8cb8-serviceca\") pod \"node-ca-zqs6v\" (UID: \"daf6535c-7f33-407a-a5d9-e1c9525c8cb8\") " pod="openshift-image-registry/node-ca-zqs6v"
Oct 30 04:43:21.598683 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.597448    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-system-cni-dir\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:43:21.598683 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.597472    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8x7k\" (UniqueName: \"kubernetes.io/projected/3b1cae8a-12cf-4b59-85f0-a63f324fc8e1-kube-api-access-f8x7k\") pod \"network-check-target-dgsdt\" (UID: \"3b1cae8a-12cf-4b59-85f0-a63f324fc8e1\") " pod="openshift-network-diagnostics/network-check-target-dgsdt"
Oct 30 04:43:21.598683 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.597498    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l294n\" (UniqueName: \"kubernetes.io/projected/bff48d36-3909-4838-b499-66e5081bcc90-kube-api-access-l294n\") pod \"iptables-alerter-st55p\" (UID: \"bff48d36-3909-4838-b499-66e5081bcc90\") " pod="openshift-network-operator/iptables-alerter-st55p"
Oct 30 04:43:21.598683 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.597524    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-multus-socket-dir-parent\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:43:21.598683 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.597548    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-host-run-k8s-cni-cncf-io\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:43:21.598683 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.597583    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-host-var-lib-cni-multus\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:43:21.598683 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.597607    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/70169887-cc72-4fb5-96b5-e09e260d7a8b-env-overrides\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:21.598683 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.597632    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pknnw\" (UniqueName: \"kubernetes.io/projected/70169887-cc72-4fb5-96b5-e09e260d7a8b-kube-api-access-pknnw\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:21.598683 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.597657    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/41a4ec4b-b407-4316-9238-dee6be432d3c-original-pull-secret\") pod \"global-pull-secret-syncer-tt5mj\" (UID: \"41a4ec4b-b407-4316-9238-dee6be432d3c\") " pod="kube-system/global-pull-secret-syncer-tt5mj"
Oct 30 04:43:21.598683 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.597683    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2a3eb4b-a128-4649-bf77-78df2705f147-cnibin\") pod \"multus-additional-cni-plugins-tnwc6\" (UID: \"a2a3eb4b-a128-4649-bf77-78df2705f147\") " pod="openshift-multus/multus-additional-cni-plugins-tnwc6"
Oct 30 04:43:21.598683 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.597708    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-multus-cni-dir\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:43:21.598683 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.597742    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/885d476c-7568-4b63-ab9d-1da7608706f5-tmp-dir\") pod \"node-resolver-gndxs\" (UID: \"885d476c-7568-4b63-ab9d-1da7608706f5\") " pod="openshift-dns/node-resolver-gndxs"
Oct 30 04:43:21.598683 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.597768    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-host-run-ovn-kubernetes\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:21.598683 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.597791    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/672524df-2f52-455b-9423-a58f53c31129-registration-dir\") pod \"aws-ebs-csi-driver-node-vhsc7\" (UID: \"672524df-2f52-455b-9423-a58f53c31129\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhsc7"
Oct 30 04:43:21.598683 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.597815    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/daf6535c-7f33-407a-a5d9-e1c9525c8cb8-host\") pod \"node-ca-zqs6v\" (UID: \"daf6535c-7f33-407a-a5d9-e1c9525c8cb8\") " pod="openshift-image-registry/node-ca-zqs6v"
Oct 30 04:43:21.598683 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.597840    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-host-run-netns\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:21.611636 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.611614    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 04:43:21.611810 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.611774    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 04:43:21.612141 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.612121    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 04:43:21.612336 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.612298    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-tdqqv\""
Oct 30 04:43:21.612336 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.612327    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 04:43:21.614431 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.612977    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 04:43:21.614431 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.613119    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bngnh\""
Oct 30 04:43:21.614431 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.613149    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 04:43:21.614431 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.613225    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 04:43:21.614431 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.613257    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 04:43:21.614431 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.613313    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tg42c\""
Oct 30 04:43:21.614431 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.613354    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 04:43:21.614431 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.613390    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-q24mq\""
Oct 30 04:43:21.614431 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.613411    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 04:43:21.614431 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.613494    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 04:43:21.614431 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.613524    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 04:43:21.614431 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.613556    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 04:43:21.614431 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.613579    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 04:43:21.614431 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.614031    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bpk6m\""
Oct 30 04:43:21.614431 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.614089    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 04:43:21.615890 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.614634    2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 04:43:21.615890 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.614694    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-g6cxg"
Oct 30 04:43:21.698162 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.698129    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-etc-sysctl-d\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:43:21.698329 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.698168    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29k58\" (UniqueName: \"kubernetes.io/projected/daf6535c-7f33-407a-a5d9-e1c9525c8cb8-kube-api-access-29k58\") pod \"node-ca-zqs6v\" (UID: \"daf6535c-7f33-407a-a5d9-e1c9525c8cb8\") " pod="openshift-image-registry/node-ca-zqs6v"
Oct 30 04:43:21.698329 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.698194    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-node-log\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:21.698329 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.698231    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-node-log\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:21.698329 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.698264    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-etc-modprobe-d\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:43:21.698329 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.698283    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-etc-sysconfig\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:43:21.698329 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.698297    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-host-slash\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:21.698329 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.698316    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/70169887-cc72-4fb5-96b5-e09e260d7a8b-ovnkube-script-lib\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:21.698670 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.698335    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bff48d36-3909-4838-b499-66e5081bcc90-host-slash\") pod \"iptables-alerter-st55p\" (UID: \"bff48d36-3909-4838-b499-66e5081bcc90\") " pod="openshift-network-operator/iptables-alerter-st55p"
Oct 30 04:43:21.698670 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.698336    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-etc-sysctl-d\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:43:21.698670 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.698351    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49nsv\" (UniqueName: \"kubernetes.io/projected/7f75d879-5e31-41fd-838c-b1f0408cb45a-kube-api-access-49nsv\") pod \"network-metrics-daemon-vjfrn\" (UID: \"7f75d879-5e31-41fd-838c-b1f0408cb45a\") " pod="openshift-multus/network-metrics-daemon-vjfrn"
Oct 30 04:43:21.698670 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.698365    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-etc-tuned\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:43:21.698670 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.698379    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/672524df-2f52-455b-9423-a58f53c31129-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vhsc7\" (UID: \"672524df-2f52-455b-9423-a58f53c31129\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhsc7"
Oct 30 04:43:21.698670 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.698383    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-host-slash\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:21.698670 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.698394    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-host-var-lib-kubelet\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:43:21.698670 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.698411    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w8n4\" (UniqueName: \"kubernetes.io/projected/bf49335f-7b8e-4212-8777-1575cd0929f1-kube-api-access-4w8n4\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:43:21.698670 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.698428    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-etc-openvswitch\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:21.698670 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.698445    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bff48d36-3909-4838-b499-66e5081bcc90-iptables-alerter-script\") pod \"iptables-alerter-st55p\" (UID: \"bff48d36-3909-4838-b499-66e5081bcc90\") " pod="openshift-network-operator/iptables-alerter-st55p"
Oct 30 04:43:21.698670 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.698452    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-etc-modprobe-d\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:43:21.698670 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.698469    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/41a4ec4b-b407-4316-9238-dee6be432d3c-dbus\") pod \"global-pull-secret-syncer-tt5mj\" (UID: \"41a4ec4b-b407-4316-9238-dee6be432d3c\") " pod="kube-system/global-pull-secret-syncer-tt5mj"
Oct 30 04:43:21.698670 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.698483    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-etc-sysctl-conf\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:43:21.698670 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.698486    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-etc-sysconfig\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:43:21.698670 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.698509    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-host-var-lib-kubelet\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:43:21.698670 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.698519    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/672524df-2f52-455b-9423-a58f53c31129-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vhsc7\" (UID: \"672524df-2f52-455b-9423-a58f53c31129\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhsc7"
Oct 30 04:43:21.698670 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.698550    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-run\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:43:21.699140 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.698574    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/672524df-2f52-455b-9423-a58f53c31129-socket-dir\") pod \"aws-ebs-csi-driver-node-vhsc7\" (UID: \"672524df-2f52-455b-9423-a58f53c31129\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhsc7"
Oct 30 04:43:21.699140 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.698590    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-etc-sysctl-conf\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:43:21.699140 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.698599    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a2a3eb4b-a128-4649-bf77-78df2705f147-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tnwc6\" (UID: \"a2a3eb4b-a128-4649-bf77-78df2705f147\") " pod="openshift-multus/multus-additional-cni-plugins-tnwc6"
Oct 30 04:43:21.699140 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.698627    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a2a3eb4b-a128-4649-bf77-78df2705f147-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tnwc6\" (UID: \"a2a3eb4b-a128-4649-bf77-78df2705f147\") " pod="openshift-multus/multus-additional-cni-plugins-tnwc6"
Oct 30 04:43:21.699140 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.698686    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-cnibin\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:43:21.699140 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.698689    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-etc-openvswitch\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:21.699140 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.698711    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-var-lib-openvswitch\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:21.699140 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.698737    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-host-cni-netd\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:21.699140 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.698764    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-etc-systemd\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:43:21.699140 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.698788    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/672524df-2f52-455b-9423-a58f53c31129-device-dir\") pod \"aws-ebs-csi-driver-node-vhsc7\" (UID: \"672524df-2f52-455b-9423-a58f53c31129\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhsc7"
Oct 30 04:43:21.699140 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.698798    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bff48d36-3909-4838-b499-66e5081bcc90-host-slash\") pod \"iptables-alerter-st55p\" (UID: \"bff48d36-3909-4838-b499-66e5081bcc90\") " pod="openshift-network-operator/iptables-alerter-st55p"
Oct 30 04:43:21.699140 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.698813    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfwq8\" (UniqueName: \"kubernetes.io/projected/672524df-2f52-455b-9423-a58f53c31129-kube-api-access-bfwq8\") pod \"aws-ebs-csi-driver-node-vhsc7\" (UID: \"672524df-2f52-455b-9423-a58f53c31129\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhsc7"
Oct 30 04:43:21.699140 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.698836    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/73a78a0d-d19e-4a96-b8ce-253d631da703-agent-certs\") pod \"konnectivity-agent-6bpl4\" (UID: \"73a78a0d-d19e-4a96-b8ce-253d631da703\") " pod="kube-system/konnectivity-agent-6bpl4"
Oct 30 04:43:21.699140 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.698855    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-run-openvswitch\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:21.699140 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.698876    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-log-socket\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:21.699140 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.698913    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/70169887-cc72-4fb5-96b5-e09e260d7a8b-ovnkube-config\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:21.699140 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.698927    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/41a4ec4b-b407-4316-9238-dee6be432d3c-dbus\") pod \"global-pull-secret-syncer-tt5mj\" (UID: \"41a4ec4b-b407-4316-9238-dee6be432d3c\") " pod="kube-system/global-pull-secret-syncer-tt5mj"
Oct 30 04:43:21.699601 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.698939    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bf49335f-7b8e-4212-8777-1575cd0929f1-multus-daemon-config\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:43:21.699601 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.698963    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-var-lib-kubelet\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:43:21.699601 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.698985    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/daf6535c-7f33-407a-a5d9-e1c9525c8cb8-serviceca\") pod \"node-ca-zqs6v\" (UID: \"daf6535c-7f33-407a-a5d9-e1c9525c8cb8\") " pod="openshift-image-registry/node-ca-zqs6v"
Oct 30 04:43:21.699601 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.698994    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-host-cni-netd\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:21.699601 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.699008    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-system-cni-dir\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:43:21.699601 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.699032    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8x7k\" (UniqueName: \"kubernetes.io/projected/3b1cae8a-12cf-4b59-85f0-a63f324fc8e1-kube-api-access-f8x7k\") pod \"network-check-target-dgsdt\" (UID: \"3b1cae8a-12cf-4b59-85f0-a63f324fc8e1\") " pod="openshift-network-diagnostics/network-check-target-dgsdt"
Oct 30 04:43:21.699601 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.699058    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l294n\" (UniqueName: \"kubernetes.io/projected/bff48d36-3909-4838-b499-66e5081bcc90-kube-api-access-l294n\") pod \"iptables-alerter-st55p\" (UID: \"bff48d36-3909-4838-b499-66e5081bcc90\") " pod="openshift-network-operator/iptables-alerter-st55p"
Oct 30 04:43:21.699601 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.699081    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-multus-socket-dir-parent\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:43:21.699601 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.699105    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-host-run-k8s-cni-cncf-io\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:43:21.699601 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.699128    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-host-var-lib-cni-multus\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:43:21.699601 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.699151    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/70169887-cc72-4fb5-96b5-e09e260d7a8b-env-overrides\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:21.699601 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.699174    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pknnw\" (UniqueName: \"kubernetes.io/projected/70169887-cc72-4fb5-96b5-e09e260d7a8b-kube-api-access-pknnw\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:21.699601 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.699201    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/41a4ec4b-b407-4316-9238-dee6be432d3c-original-pull-secret\") pod \"global-pull-secret-syncer-tt5mj\" (UID: \"41a4ec4b-b407-4316-9238-dee6be432d3c\") " pod="kube-system/global-pull-secret-syncer-tt5mj"
Oct 30 04:43:21.699601 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.699223    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2a3eb4b-a128-4649-bf77-78df2705f147-cnibin\") pod \"multus-additional-cni-plugins-tnwc6\" (UID: \"a2a3eb4b-a128-4649-bf77-78df2705f147\") " pod="openshift-multus/multus-additional-cni-plugins-tnwc6"
Oct 30 04:43:21.699601 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.699247    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-multus-cni-dir\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:43:21.699601 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.699273    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/885d476c-7568-4b63-ab9d-1da7608706f5-tmp-dir\") pod \"node-resolver-gndxs\" (UID: \"885d476c-7568-4b63-ab9d-1da7608706f5\") " pod="openshift-dns/node-resolver-gndxs"
Oct 30 04:43:21.699601 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.699272    2579 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 04:43:21.699601 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.699296    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-host-run-ovn-kubernetes\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:21.700172 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.699317    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/672524df-2f52-455b-9423-a58f53c31129-registration-dir\") pod \"aws-ebs-csi-driver-node-vhsc7\" (UID: \"672524df-2f52-455b-9423-a58f53c31129\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhsc7"
Oct 30 04:43:21.700172 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.699337    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/daf6535c-7f33-407a-a5d9-e1c9525c8cb8-host\") pod \"node-ca-zqs6v\" (UID: \"daf6535c-7f33-407a-a5d9-e1c9525c8cb8\") " pod="openshift-image-registry/node-ca-zqs6v"
Oct 30 04:43:21.700172 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.699355    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-host-run-netns\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:21.700172 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.699375    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/672524df-2f52-455b-9423-a58f53c31129-etc-selinux\") pod \"aws-ebs-csi-driver-node-vhsc7\" (UID: \"672524df-2f52-455b-9423-a58f53c31129\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhsc7"
Oct 30 04:43:21.700172 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.699396    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-host-run-multus-certs\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:43:21.700172 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.699416    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-lib-modules\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:43:21.700172 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.699435    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/672524df-2f52-455b-9423-a58f53c31129-sys-fs\") pod \"aws-ebs-csi-driver-node-vhsc7\" (UID: \"672524df-2f52-455b-9423-a58f53c31129\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhsc7"
Oct 30 04:43:21.700172 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.699455    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-systemd-units\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:21.700172 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.699477    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-run-systemd\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:21.700172 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.699508    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f75d879-5e31-41fd-838c-b1f0408cb45a-metrics-certs\") pod \"network-metrics-daemon-vjfrn\" (UID: \"7f75d879-5e31-41fd-838c-b1f0408cb45a\") " pod="openshift-multus/network-metrics-daemon-vjfrn"
Oct 30 04:43:21.700172 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.699540    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/41a4ec4b-b407-4316-9238-dee6be432d3c-kubelet-config\") pod \"global-pull-secret-syncer-tt5mj\" (UID: \"41a4ec4b-b407-4316-9238-dee6be432d3c\") " pod="kube-system/global-pull-secret-syncer-tt5mj"
Oct 30 04:43:21.700172 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.699568    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-host-run-netns\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:43:21.700172 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.699589    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-host-cni-bin\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:21.700172 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.699611    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/885d476c-7568-4b63-ab9d-1da7608706f5-hosts-file\") pod \"node-resolver-gndxs\" (UID: \"885d476c-7568-4b63-ab9d-1da7608706f5\") " pod="openshift-dns/node-resolver-gndxs"
Oct 30 04:43:21.700172 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.699634    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2a3eb4b-a128-4649-bf77-78df2705f147-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tnwc6\" (UID: \"a2a3eb4b-a128-4649-bf77-78df2705f147\") " pod="openshift-multus/multus-additional-cni-plugins-tnwc6"
Oct 30 04:43:21.700172 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.699650    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-sys\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:43:21.700172 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.699665    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-tmp\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:43:21.700880 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.699680    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2a3eb4b-a128-4649-bf77-78df2705f147-system-cni-dir\") pod \"multus-additional-cni-plugins-tnwc6\" (UID: \"a2a3eb4b-a128-4649-bf77-78df2705f147\") " pod="openshift-multus/multus-additional-cni-plugins-tnwc6"
Oct 30 04:43:21.700880 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.699696    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a2a3eb4b-a128-4649-bf77-78df2705f147-cni-binary-copy\") pod \"multus-additional-cni-plugins-tnwc6\" (UID: \"a2a3eb4b-a128-4649-bf77-78df2705f147\") " pod="openshift-multus/multus-additional-cni-plugins-tnwc6"
Oct 30 04:43:21.700880 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.699711    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/73a78a0d-d19e-4a96-b8ce-253d631da703-konnectivity-ca\") pod \"konnectivity-agent-6bpl4\" (UID: \"73a78a0d-d19e-4a96-b8ce-253d631da703\") " pod="kube-system/konnectivity-agent-6bpl4"
Oct 30 04:43:21.700880 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.699725    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-run-ovn\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:21.700880 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.699739    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-host-kubelet\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:21.700880 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.699762    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-etc-kubernetes\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:43:21.700880 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.699776    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-hostroot\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:43:21.700880 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.699794    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-multus-conf-dir\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:43:21.700880 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.699810    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:21.700880 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.699825    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-etc-kubernetes\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:43:21.700880 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.699843    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwcw4\" (UniqueName: \"kubernetes.io/projected/885d476c-7568-4b63-ab9d-1da7608706f5-kube-api-access-kwcw4\") pod \"node-resolver-gndxs\" (UID: \"885d476c-7568-4b63-ab9d-1da7608706f5\") " pod="openshift-dns/node-resolver-gndxs"
Oct 30 04:43:21.700880 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.699858    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxpck\" (UniqueName: \"kubernetes.io/projected/a2a3eb4b-a128-4649-bf77-78df2705f147-kube-api-access-nxpck\") pod \"multus-additional-cni-plugins-tnwc6\" (UID: \"a2a3eb4b-a128-4649-bf77-78df2705f147\") " pod="openshift-multus/multus-additional-cni-plugins-tnwc6"
Oct 30 04:43:21.700880 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.699873    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bf49335f-7b8e-4212-8777-1575cd0929f1-cni-binary-copy\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:43:21.700880 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.699889    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/70169887-cc72-4fb5-96b5-e09e260d7a8b-ovn-node-metrics-cert\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:21.700880 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.699922    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-host\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:43:21.700880 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.699947    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92qd4\" (UniqueName: \"kubernetes.io/projected/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-kube-api-access-92qd4\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:43:21.700880 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.699975    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-host-run-netns\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:21.701634 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.699977    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-host-var-lib-cni-bin\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:43:21.701634 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.699999    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2a3eb4b-a128-4649-bf77-78df2705f147-system-cni-dir\") pod \"multus-additional-cni-plugins-tnwc6\" (UID: \"a2a3eb4b-a128-4649-bf77-78df2705f147\") " pod="openshift-multus/multus-additional-cni-plugins-tnwc6"
Oct 30 04:43:21.701634 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.700012    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bff48d36-3909-4838-b499-66e5081bcc90-iptables-alerter-script\") pod \"iptables-alerter-st55p\" (UID: \"bff48d36-3909-4838-b499-66e5081bcc90\") " pod="openshift-network-operator/iptables-alerter-st55p"
Oct 30 04:43:21.701634 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.700022    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:21.701634 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.700047    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/672524df-2f52-455b-9423-a58f53c31129-socket-dir\") pod \"aws-ebs-csi-driver-node-vhsc7\" (UID: \"672524df-2f52-455b-9423-a58f53c31129\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhsc7"
Oct 30 04:43:21.701634 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.700078    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-run-systemd\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:21.701634 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.700091    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-host-kubelet\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:21.701634 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.700110    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2a3eb4b-a128-4649-bf77-78df2705f147-cnibin\") pod \"multus-additional-cni-plugins-tnwc6\" (UID: \"a2a3eb4b-a128-4649-bf77-78df2705f147\") " pod="openshift-multus/multus-additional-cni-plugins-tnwc6"
Oct 30 04:43:21.701634 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.700127    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-multus-conf-dir\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:43:21.701634 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.700139    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/41a4ec4b-b407-4316-9238-dee6be432d3c-kubelet-config\") pod \"global-pull-secret-syncer-tt5mj\" (UID: \"41a4ec4b-b407-4316-9238-dee6be432d3c\") " pod="kube-system/global-pull-secret-syncer-tt5mj"
Oct 30 04:43:21.701634 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.700154    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-etc-kubernetes\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:43:21.701634 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.700173    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/672524df-2f52-455b-9423-a58f53c31129-device-dir\") pod \"aws-ebs-csi-driver-node-vhsc7\" (UID: \"672524df-2f52-455b-9423-a58f53c31129\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhsc7"
Oct 30 04:43:21.701634 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.700176    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-host\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:43:21.701634 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.700219    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-sys\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:43:21.701634 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.700520    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-systemd-units\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:21.701634 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.700858    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-run-openvswitch\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:21.701634 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.700936    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-log-socket\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:21.702261 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.701547    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/70169887-cc72-4fb5-96b5-e09e260d7a8b-ovnkube-script-lib\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:21.702261 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.701807    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2a3eb4b-a128-4649-bf77-78df2705f147-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tnwc6\" (UID: \"a2a3eb4b-a128-4649-bf77-78df2705f147\") " pod="openshift-multus/multus-additional-cni-plugins-tnwc6"
Oct 30 04:43:21.702261 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.702222    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-var-lib-kubelet\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:43:21.702389 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.702340    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/70169887-cc72-4fb5-96b5-e09e260d7a8b-ovnkube-config\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:21.702443 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.702392    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-system-cni-dir\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:43:21.702485 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:21.702459    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 04:43:21.702485 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.702464    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-host-run-ovn-kubernetes\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:21.703401 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.702589    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/672524df-2f52-455b-9423-a58f53c31129-registration-dir\") pod \"aws-ebs-csi-driver-node-vhsc7\" (UID: \"672524df-2f52-455b-9423-a58f53c31129\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhsc7"
Oct 30 04:43:21.703401 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.702718    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/daf6535c-7f33-407a-a5d9-e1c9525c8cb8-host\") pod \"node-ca-zqs6v\" (UID: \"daf6535c-7f33-407a-a5d9-e1c9525c8cb8\") " pod="openshift-image-registry/node-ca-zqs6v"
Oct 30 04:43:21.703401 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.702788    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/daf6535c-7f33-407a-a5d9-e1c9525c8cb8-serviceca\") pod \"node-ca-zqs6v\" (UID: \"daf6535c-7f33-407a-a5d9-e1c9525c8cb8\") " pod="openshift-image-registry/node-ca-zqs6v"
Oct 30 04:43:21.703401 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.702796    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/672524df-2f52-455b-9423-a58f53c31129-etc-selinux\") pod \"aws-ebs-csi-driver-node-vhsc7\" (UID: \"672524df-2f52-455b-9423-a58f53c31129\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhsc7"
Oct 30 04:43:21.703401 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.703385    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-multus-socket-dir-parent\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:43:21.703401 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.703395    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-etc-kubernetes\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:43:21.704310 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.703477    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/885d476c-7568-4b63-ab9d-1da7608706f5-hosts-file\") pod \"node-resolver-gndxs\" (UID: \"885d476c-7568-4b63-ab9d-1da7608706f5\") " pod="openshift-dns/node-resolver-gndxs"
Oct 30 04:43:21.704310 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.703955    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-var-lib-openvswitch\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:21.704389 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.704312    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-host-run-k8s-cni-cncf-io\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:43:21.704628 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.704608    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/885d476c-7568-4b63-ab9d-1da7608706f5-tmp-dir\") pod \"node-resolver-gndxs\" (UID: \"885d476c-7568-4b63-ab9d-1da7608706f5\") " pod="openshift-dns/node-resolver-gndxs"
Oct 30 04:43:21.704837 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.704813    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-etc-systemd\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:43:21.704998 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.704875    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-hostroot\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:43:21.704998 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.704950    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-host-run-netns\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:43:21.704998 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.704875    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bf49335f-7b8e-4212-8777-1575cd0929f1-multus-daemon-config\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:43:21.704998 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.704951    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-multus-cni-dir\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:43:21.704998 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.704993    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-host-var-lib-cni-multus\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:43:21.705310 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.705066    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/672524df-2f52-455b-9423-a58f53c31129-sys-fs\") pod \"aws-ebs-csi-driver-node-vhsc7\" (UID: \"672524df-2f52-455b-9423-a58f53c31129\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhsc7"
Oct 30 04:43:21.705310 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.705095    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-lib-modules\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:43:21.705310 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.705145    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-host-run-multus-certs\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:43:21.705310 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.705195    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-cnibin\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:43:21.705446 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:21.705342    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f75d879-5e31-41fd-838c-b1f0408cb45a-metrics-certs podName:7f75d879-5e31-41fd-838c-b1f0408cb45a nodeName:}" failed. No retries permitted until 2026-10-30 04:43:22.205317472 +0000 UTC m=+3.137283195 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7f75d879-5e31-41fd-838c-b1f0408cb45a-metrics-certs") pod "network-metrics-daemon-vjfrn" (UID: "7f75d879-5e31-41fd-838c-b1f0408cb45a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 04:43:21.707297 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.705799    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a2a3eb4b-a128-4649-bf77-78df2705f147-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tnwc6\" (UID: \"a2a3eb4b-a128-4649-bf77-78df2705f147\") " pod="openshift-multus/multus-additional-cni-plugins-tnwc6"
Oct 30 04:43:21.707297 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.705875    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-run-ovn\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:21.707297 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.705883    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bf49335f-7b8e-4212-8777-1575cd0929f1-cni-binary-copy\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:43:21.707297 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.705933    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/70169887-cc72-4fb5-96b5-e09e260d7a8b-env-overrides\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:21.707297 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.705945    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-host-cni-bin\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:21.707297 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:21.706013    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 04:43:21.707297 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.706046    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-run\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:43:21.707297 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:21.706085    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41a4ec4b-b407-4316-9238-dee6be432d3c-original-pull-secret podName:41a4ec4b-b407-4316-9238-dee6be432d3c nodeName:}" failed. No retries permitted until 2026-10-30 04:43:22.206066946 +0000 UTC m=+3.138032651 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/41a4ec4b-b407-4316-9238-dee6be432d3c-original-pull-secret") pod "global-pull-secret-syncer-tt5mj" (UID: "41a4ec4b-b407-4316-9238-dee6be432d3c") : object "kube-system"/"original-pull-secret" not registered
Oct 30 04:43:21.707297 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.706011    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a2a3eb4b-a128-4649-bf77-78df2705f147-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tnwc6\" (UID: \"a2a3eb4b-a128-4649-bf77-78df2705f147\") " pod="openshift-multus/multus-additional-cni-plugins-tnwc6"
Oct 30 04:43:21.707297 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.706116    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-host-var-lib-cni-bin\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:43:21.708960 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.707940    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/73a78a0d-d19e-4a96-b8ce-253d631da703-konnectivity-ca\") pod \"konnectivity-agent-6bpl4\" (UID: \"73a78a0d-d19e-4a96-b8ce-253d631da703\") " pod="kube-system/konnectivity-agent-6bpl4"
Oct 30 04:43:21.708960 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.707945    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a2a3eb4b-a128-4649-bf77-78df2705f147-cni-binary-copy\") pod \"multus-additional-cni-plugins-tnwc6\" (UID: \"a2a3eb4b-a128-4649-bf77-78df2705f147\") " pod="openshift-multus/multus-additional-cni-plugins-tnwc6"
Oct 30 04:43:21.709183 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.709090    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-etc-tuned\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:43:21.709343 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.709324    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-tmp\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:43:21.717392 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.717370    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/70169887-cc72-4fb5-96b5-e09e260d7a8b-ovn-node-metrics-cert\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:21.717565 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.717511    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/73a78a0d-d19e-4a96-b8ce-253d631da703-agent-certs\") pod \"konnectivity-agent-6bpl4\" (UID: \"73a78a0d-d19e-4a96-b8ce-253d631da703\") " pod="kube-system/konnectivity-agent-6bpl4"
Oct 30 04:43:21.721127 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:21.721104    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 04:43:21.721240 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:21.721131    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 04:43:21.721240 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:21.721146    2579 projected.go:196] Error preparing data for projected volume kube-api-access-f8x7k for pod openshift-network-diagnostics/network-check-target-dgsdt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 04:43:21.721240 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:21.721219    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b1cae8a-12cf-4b59-85f0-a63f324fc8e1-kube-api-access-f8x7k podName:3b1cae8a-12cf-4b59-85f0-a63f324fc8e1 nodeName:}" failed. No retries permitted until 2026-10-30 04:43:22.221201107 +0000 UTC m=+3.153166829 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-f8x7k" (UniqueName: "kubernetes.io/projected/3b1cae8a-12cf-4b59-85f0-a63f324fc8e1-kube-api-access-f8x7k") pod "network-check-target-dgsdt" (UID: "3b1cae8a-12cf-4b59-85f0-a63f324fc8e1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 04:43:21.723408 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.723380    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfwq8\" (UniqueName: \"kubernetes.io/projected/672524df-2f52-455b-9423-a58f53c31129-kube-api-access-bfwq8\") pod \"aws-ebs-csi-driver-node-vhsc7\" (UID: \"672524df-2f52-455b-9423-a58f53c31129\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhsc7"
Oct 30 04:43:21.723611 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.723592    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49nsv\" (UniqueName: \"kubernetes.io/projected/7f75d879-5e31-41fd-838c-b1f0408cb45a-kube-api-access-49nsv\") pod \"network-metrics-daemon-vjfrn\" (UID: \"7f75d879-5e31-41fd-838c-b1f0408cb45a\") " pod="openshift-multus/network-metrics-daemon-vjfrn"
Oct 30 04:43:21.723735 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.723604    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxpck\" (UniqueName: \"kubernetes.io/projected/a2a3eb4b-a128-4649-bf77-78df2705f147-kube-api-access-nxpck\") pod \"multus-additional-cni-plugins-tnwc6\" (UID: \"a2a3eb4b-a128-4649-bf77-78df2705f147\") " pod="openshift-multus/multus-additional-cni-plugins-tnwc6"
Oct 30 04:43:21.724004 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.723984    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92qd4\" (UniqueName: \"kubernetes.io/projected/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-kube-api-access-92qd4\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:43:21.724162 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.724136    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w8n4\" (UniqueName: \"kubernetes.io/projected/bf49335f-7b8e-4212-8777-1575cd0929f1-kube-api-access-4w8n4\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:43:21.725784 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.725737    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pknnw\" (UniqueName: \"kubernetes.io/projected/70169887-cc72-4fb5-96b5-e09e260d7a8b-kube-api-access-pknnw\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:21.725924 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.725884    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwcw4\" (UniqueName: \"kubernetes.io/projected/885d476c-7568-4b63-ab9d-1da7608706f5-kube-api-access-kwcw4\") pod \"node-resolver-gndxs\" (UID: \"885d476c-7568-4b63-ab9d-1da7608706f5\") " pod="openshift-dns/node-resolver-gndxs"
Oct 30 04:43:21.726955 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.726936    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l294n\" (UniqueName: \"kubernetes.io/projected/bff48d36-3909-4838-b499-66e5081bcc90-kube-api-access-l294n\") pod \"iptables-alerter-st55p\" (UID: \"bff48d36-3909-4838-b499-66e5081bcc90\") " pod="openshift-network-operator/iptables-alerter-st55p"
Oct 30 04:43:21.727836 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.727796    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29k58\" (UniqueName: \"kubernetes.io/projected/daf6535c-7f33-407a-a5d9-e1c9525c8cb8-kube-api-access-29k58\") pod \"node-ca-zqs6v\" (UID: \"daf6535c-7f33-407a-a5d9-e1c9525c8cb8\") " pod="openshift-image-registry/node-ca-zqs6v"
Oct 30 04:43:21.806483 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.806449    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zqs6v"
Oct 30 04:43:21.817873 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.817846    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6bpl4"
Oct 30 04:43:21.822951 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:21.822927    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaf6535c_7f33_407a_a5d9_e1c9525c8cb8.slice/crio-f7cadb24483929c3c6476697bf27ec2f6bcb8b095253c00d16ea594659edc7da WatchSource:0}: Error finding container f7cadb24483929c3c6476697bf27ec2f6bcb8b095253c00d16ea594659edc7da: Status 404 returned error can't find the container with id f7cadb24483929c3c6476697bf27ec2f6bcb8b095253c00d16ea594659edc7da
Oct 30 04:43:21.828284 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:21.828258    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73a78a0d_d19e_4a96_b8ce_253d631da703.slice/crio-2337701261d2bc376ff506ce1806be3ee7b49efec319b3e1b59e756b19f4f424 WatchSource:0}: Error finding container 2337701261d2bc376ff506ce1806be3ee7b49efec319b3e1b59e756b19f4f424: Status 404 returned error can't find the container with id 2337701261d2bc376ff506ce1806be3ee7b49efec319b3e1b59e756b19f4f424
Oct 30 04:43:21.830043 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.830014    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:43:21.834939 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.834919    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-c9tpc"
Oct 30 04:43:21.846528 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.846494    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhsc7"
Oct 30 04:43:21.851029 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.851012    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gndxs"
Oct 30 04:43:21.870417 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:21.870391    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf49335f_7b8e_4212_8777_1575cd0929f1.slice/crio-2e751d48664d410a227b1bec20c5227213cafb620e21361743051bb9854998f7 WatchSource:0}: Error finding container 2e751d48664d410a227b1bec20c5227213cafb620e21361743051bb9854998f7: Status 404 returned error can't find the container with id 2e751d48664d410a227b1bec20c5227213cafb620e21361743051bb9854998f7
Oct 30 04:43:21.872069 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:21.871957    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff3a5b12_bdd2_47d1_9475_ee3b2bcba935.slice/crio-0ceada87957df9cf161eb9f911c7a041fcb6651c7fcd55f543b6184737b951ee WatchSource:0}: Error finding container 0ceada87957df9cf161eb9f911c7a041fcb6651c7fcd55f543b6184737b951ee: Status 404 returned error can't find the container with id 0ceada87957df9cf161eb9f911c7a041fcb6651c7fcd55f543b6184737b951ee
Oct 30 04:43:21.874525 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:21.874494    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod672524df_2f52_455b_9423_a58f53c31129.slice/crio-7b4b0fdfa213c0d6369f78967e3e637a533d14c38e5ea2c175afde0913212db3 WatchSource:0}: Error finding container 7b4b0fdfa213c0d6369f78967e3e637a533d14c38e5ea2c175afde0913212db3: Status 404 returned error can't find the container with id 7b4b0fdfa213c0d6369f78967e3e637a533d14c38e5ea2c175afde0913212db3
Oct 30 04:43:21.878638 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.878620    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-st55p"
Oct 30 04:43:21.883332 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.883306    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:21.891401 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:21.891380    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tnwc6"
Oct 30 04:43:21.891560 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:21.891536    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbff48d36_3909_4838_b499_66e5081bcc90.slice/crio-9ed996658a95d487489d306eef164e7090b0c202b685ef01b49e6ac9d768be91 WatchSource:0}: Error finding container 9ed996658a95d487489d306eef164e7090b0c202b685ef01b49e6ac9d768be91: Status 404 returned error can't find the container with id 9ed996658a95d487489d306eef164e7090b0c202b685ef01b49e6ac9d768be91
Oct 30 04:43:21.895255 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:21.895233    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70169887_cc72_4fb5_96b5_e09e260d7a8b.slice/crio-52237db8a2daea675ec54c056e14360e3eced11310ea7b7a06f9771cbc8fa81d WatchSource:0}: Error finding container 52237db8a2daea675ec54c056e14360e3eced11310ea7b7a06f9771cbc8fa81d: Status 404 returned error can't find the container with id 52237db8a2daea675ec54c056e14360e3eced11310ea7b7a06f9771cbc8fa81d
Oct 30 04:43:21.900983 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:43:21.900961    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2a3eb4b_a128_4649_bf77_78df2705f147.slice/crio-622b5622e07128e7fe97898d1e42bf52701f806897f41e5c1599820de4a78858 WatchSource:0}: Error finding container 622b5622e07128e7fe97898d1e42bf52701f806897f41e5c1599820de4a78858: Status 404 returned error can't find the container with id 622b5622e07128e7fe97898d1e42bf52701f806897f41e5c1599820de4a78858
Oct 30 04:43:22.304561 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:22.304528    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/41a4ec4b-b407-4316-9238-dee6be432d3c-original-pull-secret\") pod \"global-pull-secret-syncer-tt5mj\" (UID: \"41a4ec4b-b407-4316-9238-dee6be432d3c\") " pod="kube-system/global-pull-secret-syncer-tt5mj"
Oct 30 04:43:22.304742 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:22.304571    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f75d879-5e31-41fd-838c-b1f0408cb45a-metrics-certs\") pod \"network-metrics-daemon-vjfrn\" (UID: \"7f75d879-5e31-41fd-838c-b1f0408cb45a\") " pod="openshift-multus/network-metrics-daemon-vjfrn"
Oct 30 04:43:22.304742 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:22.304624    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8x7k\" (UniqueName: \"kubernetes.io/projected/3b1cae8a-12cf-4b59-85f0-a63f324fc8e1-kube-api-access-f8x7k\") pod \"network-check-target-dgsdt\" (UID: \"3b1cae8a-12cf-4b59-85f0-a63f324fc8e1\") " pod="openshift-network-diagnostics/network-check-target-dgsdt"
Oct 30 04:43:22.304858 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:22.304761    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 04:43:22.304858 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:22.304781    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 04:43:22.304858 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:22.304794    2579 projected.go:196] Error preparing data for projected volume kube-api-access-f8x7k for pod openshift-network-diagnostics/network-check-target-dgsdt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 04:43:22.304858 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:22.304851    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b1cae8a-12cf-4b59-85f0-a63f324fc8e1-kube-api-access-f8x7k podName:3b1cae8a-12cf-4b59-85f0-a63f324fc8e1 nodeName:}" failed. No retries permitted until 2026-10-30 04:43:23.30483323 +0000 UTC m=+4.236798941 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-f8x7k" (UniqueName: "kubernetes.io/projected/3b1cae8a-12cf-4b59-85f0-a63f324fc8e1-kube-api-access-f8x7k") pod "network-check-target-dgsdt" (UID: "3b1cae8a-12cf-4b59-85f0-a63f324fc8e1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 04:43:22.305146 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:22.305127    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 04:43:22.305215 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:22.305185    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f75d879-5e31-41fd-838c-b1f0408cb45a-metrics-certs podName:7f75d879-5e31-41fd-838c-b1f0408cb45a nodeName:}" failed. No retries permitted until 2026-10-30 04:43:23.305169563 +0000 UTC m=+4.237135282 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7f75d879-5e31-41fd-838c-b1f0408cb45a-metrics-certs") pod "network-metrics-daemon-vjfrn" (UID: "7f75d879-5e31-41fd-838c-b1f0408cb45a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 04:43:22.305260 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:22.305241    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 04:43:22.305292 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:22.305271    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41a4ec4b-b407-4316-9238-dee6be432d3c-original-pull-secret podName:41a4ec4b-b407-4316-9238-dee6be432d3c nodeName:}" failed. No retries permitted until 2026-10-30 04:43:23.305260487 +0000 UTC m=+4.237226197 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/41a4ec4b-b407-4316-9238-dee6be432d3c-original-pull-secret") pod "global-pull-secret-syncer-tt5mj" (UID: "41a4ec4b-b407-4316-9238-dee6be432d3c") : object "kube-system"/"original-pull-secret" not registered
Oct 30 04:43:22.574523 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:22.574438    2579 generic.go:356] "Generic (PLEG): container finished" podID="b4286e30fd1fc24b304f69bb80b9a852" containerID="235f96ed71c0e7f50b8cc1411b837d1e00a1fc051da2365bd87ecce415d6f82d" exitCode=0
Oct 30 04:43:22.575377 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:22.574541    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-98.ec2.internal" event={"ID":"b4286e30fd1fc24b304f69bb80b9a852","Type":"ContainerDied","Data":"235f96ed71c0e7f50b8cc1411b837d1e00a1fc051da2365bd87ecce415d6f82d"}
Oct 30 04:43:22.600110 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:22.599766    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-98.ec2.internal" event={"ID":"38fe9c19c3aaab70dd96caa47ec7e827","Type":"ContainerStarted","Data":"6a8ce66c4e40fcd7b3b1c86c553bfa2fe98099b33796195cd1d8235ac79fee4e"}
Oct 30 04:43:22.613420 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:22.613391    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-st55p" event={"ID":"bff48d36-3909-4838-b499-66e5081bcc90","Type":"ContainerStarted","Data":"9ed996658a95d487489d306eef164e7090b0c202b685ef01b49e6ac9d768be91"}
Oct 30 04:43:22.617681 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:22.617578    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 04:38:21 +0000 UTC" deadline="2028-06-17 16:23:32.070510333 +0000 UTC"
Oct 30 04:43:22.617681 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:22.617633    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14315h40m9.452880834s"
Oct 30 04:43:22.624940 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:22.621889    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gndxs" event={"ID":"885d476c-7568-4b63-ab9d-1da7608706f5","Type":"ContainerStarted","Data":"fc25014589e21d741b04fdcef4af9d6eec45f39f3afd74e3b0e3f95b25e58d35"}
Oct 30 04:43:22.625433 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:22.625387    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qtb8v" event={"ID":"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935","Type":"ContainerStarted","Data":"0ceada87957df9cf161eb9f911c7a041fcb6651c7fcd55f543b6184737b951ee"}
Oct 30 04:43:22.627562 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:22.627536    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zqs6v" event={"ID":"daf6535c-7f33-407a-a5d9-e1c9525c8cb8","Type":"ContainerStarted","Data":"f7cadb24483929c3c6476697bf27ec2f6bcb8b095253c00d16ea594659edc7da"}
Oct 30 04:43:22.648783 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:22.648755    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tnwc6" event={"ID":"a2a3eb4b-a128-4649-bf77-78df2705f147","Type":"ContainerStarted","Data":"622b5622e07128e7fe97898d1e42bf52701f806897f41e5c1599820de4a78858"}
Oct 30 04:43:22.668026 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:22.667995    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv" event={"ID":"70169887-cc72-4fb5-96b5-e09e260d7a8b","Type":"ContainerStarted","Data":"52237db8a2daea675ec54c056e14360e3eced11310ea7b7a06f9771cbc8fa81d"}
Oct 30 04:43:22.682823 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:22.682791    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhsc7" event={"ID":"672524df-2f52-455b-9423-a58f53c31129","Type":"ContainerStarted","Data":"7b4b0fdfa213c0d6369f78967e3e637a533d14c38e5ea2c175afde0913212db3"}
Oct 30 04:43:22.686608 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:22.686550    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-c9tpc" event={"ID":"bf49335f-7b8e-4212-8777-1575cd0929f1","Type":"ContainerStarted","Data":"2e751d48664d410a227b1bec20c5227213cafb620e21361743051bb9854998f7"}
Oct 30 04:43:22.702935 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:22.702856    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6bpl4" event={"ID":"73a78a0d-d19e-4a96-b8ce-253d631da703","Type":"ContainerStarted","Data":"2337701261d2bc376ff506ce1806be3ee7b49efec319b3e1b59e756b19f4f424"}
Oct 30 04:43:23.312475 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:23.312437    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8x7k\" (UniqueName: \"kubernetes.io/projected/3b1cae8a-12cf-4b59-85f0-a63f324fc8e1-kube-api-access-f8x7k\") pod \"network-check-target-dgsdt\" (UID: \"3b1cae8a-12cf-4b59-85f0-a63f324fc8e1\") " pod="openshift-network-diagnostics/network-check-target-dgsdt"
Oct 30 04:43:23.312658 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:23.312492    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/41a4ec4b-b407-4316-9238-dee6be432d3c-original-pull-secret\") pod \"global-pull-secret-syncer-tt5mj\" (UID: \"41a4ec4b-b407-4316-9238-dee6be432d3c\") " pod="kube-system/global-pull-secret-syncer-tt5mj"
Oct 30 04:43:23.312658 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:23.312526    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f75d879-5e31-41fd-838c-b1f0408cb45a-metrics-certs\") pod \"network-metrics-daemon-vjfrn\" (UID: \"7f75d879-5e31-41fd-838c-b1f0408cb45a\") " pod="openshift-multus/network-metrics-daemon-vjfrn"
Oct 30 04:43:23.312772 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:23.312658    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 04:43:23.312772 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:23.312713    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f75d879-5e31-41fd-838c-b1f0408cb45a-metrics-certs podName:7f75d879-5e31-41fd-838c-b1f0408cb45a nodeName:}" failed. No retries permitted until 2026-10-30 04:43:25.312695385 +0000 UTC m=+6.244661097 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7f75d879-5e31-41fd-838c-b1f0408cb45a-metrics-certs") pod "network-metrics-daemon-vjfrn" (UID: "7f75d879-5e31-41fd-838c-b1f0408cb45a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 04:43:23.313051 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:23.313030    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 04:43:23.313139 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:23.313065    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 04:43:23.313139 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:23.313088    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 04:43:23.313139 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:23.313092    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41a4ec4b-b407-4316-9238-dee6be432d3c-original-pull-secret podName:41a4ec4b-b407-4316-9238-dee6be432d3c nodeName:}" failed. No retries permitted until 2026-10-30 04:43:25.31307732 +0000 UTC m=+6.245043031 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/41a4ec4b-b407-4316-9238-dee6be432d3c-original-pull-secret") pod "global-pull-secret-syncer-tt5mj" (UID: "41a4ec4b-b407-4316-9238-dee6be432d3c") : object "kube-system"/"original-pull-secret" not registered
Oct 30 04:43:23.313139 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:23.313100    2579 projected.go:196] Error preparing data for projected volume kube-api-access-f8x7k for pod openshift-network-diagnostics/network-check-target-dgsdt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 04:43:23.313429 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:23.313150    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b1cae8a-12cf-4b59-85f0-a63f324fc8e1-kube-api-access-f8x7k podName:3b1cae8a-12cf-4b59-85f0-a63f324fc8e1 nodeName:}" failed. No retries permitted until 2026-10-30 04:43:25.313133105 +0000 UTC m=+6.245098827 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-f8x7k" (UniqueName: "kubernetes.io/projected/3b1cae8a-12cf-4b59-85f0-a63f324fc8e1-kube-api-access-f8x7k") pod "network-check-target-dgsdt" (UID: "3b1cae8a-12cf-4b59-85f0-a63f324fc8e1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 04:43:23.350989 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:23.350619    2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 04:43:23.556261 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:23.555407    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vjfrn"
Oct 30 04:43:23.556261 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:23.555557    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vjfrn" podUID="7f75d879-5e31-41fd-838c-b1f0408cb45a"
Oct 30 04:43:23.556261 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:23.555959    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgsdt"
Oct 30 04:43:23.556261 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:23.556043    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgsdt" podUID="3b1cae8a-12cf-4b59-85f0-a63f324fc8e1"
Oct 30 04:43:23.556261 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:23.556103    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tt5mj"
Oct 30 04:43:23.556261 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:23.556171    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tt5mj" podUID="41a4ec4b-b407-4316-9238-dee6be432d3c"
Oct 30 04:43:23.618841 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:23.618758    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 04:38:21 +0000 UTC" deadline="2028-04-20 16:10:01.861584725 +0000 UTC"
Oct 30 04:43:23.618841 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:23.618817    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12923h26m38.242789575s"
Oct 30 04:43:23.730155 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:23.730085    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-98.ec2.internal" event={"ID":"b4286e30fd1fc24b304f69bb80b9a852","Type":"ContainerStarted","Data":"fbc8a8656cf7e6b8c2e962abd1965ed6e175cb717ddb802632f59ecb6abdee2d"}
Oct 30 04:43:23.769337 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:23.769276    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-98.ec2.internal" podStartSLOduration=3.769256012 podStartE2EDuration="3.769256012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:43:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:43:22.652708623 +0000 UTC m=+3.584674350" watchObservedRunningTime="2026-10-30 04:43:23.769256012 +0000 UTC m=+4.701221740"
Oct 30 04:43:24.742674 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:24.742585    2579 generic.go:356] "Generic (PLEG): container finished" podID="a2a3eb4b-a128-4649-bf77-78df2705f147" containerID="887ebee1bedc93413e427e8634657307f14f3ecff030671b8d486d090bb2df2e" exitCode=0
Oct 30 04:43:24.743791 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:24.743658    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tnwc6" event={"ID":"a2a3eb4b-a128-4649-bf77-78df2705f147","Type":"ContainerDied","Data":"887ebee1bedc93413e427e8634657307f14f3ecff030671b8d486d090bb2df2e"}
Oct 30 04:43:24.813789 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:24.813631    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-98.ec2.internal" podStartSLOduration=4.813616168 podStartE2EDuration="4.813616168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:43:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:43:23.769916281 +0000 UTC m=+4.701882007" watchObservedRunningTime="2026-10-30 04:43:24.813616168 +0000 UTC m=+5.745581894"
Oct 30 04:43:25.329557 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:25.329509    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8x7k\" (UniqueName: \"kubernetes.io/projected/3b1cae8a-12cf-4b59-85f0-a63f324fc8e1-kube-api-access-f8x7k\") pod \"network-check-target-dgsdt\" (UID: \"3b1cae8a-12cf-4b59-85f0-a63f324fc8e1\") " pod="openshift-network-diagnostics/network-check-target-dgsdt"
Oct 30 04:43:25.329742 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:25.329570    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/41a4ec4b-b407-4316-9238-dee6be432d3c-original-pull-secret\") pod \"global-pull-secret-syncer-tt5mj\" (UID: \"41a4ec4b-b407-4316-9238-dee6be432d3c\") " pod="kube-system/global-pull-secret-syncer-tt5mj"
Oct 30 04:43:25.329742 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:25.329604    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f75d879-5e31-41fd-838c-b1f0408cb45a-metrics-certs\") pod \"network-metrics-daemon-vjfrn\" (UID: \"7f75d879-5e31-41fd-838c-b1f0408cb45a\") " pod="openshift-multus/network-metrics-daemon-vjfrn"
Oct 30 04:43:25.329742 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:25.329736    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 04:43:25.329919 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:25.329804    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f75d879-5e31-41fd-838c-b1f0408cb45a-metrics-certs podName:7f75d879-5e31-41fd-838c-b1f0408cb45a nodeName:}" failed. No retries permitted until 2026-10-30 04:43:29.329784167 +0000 UTC m=+10.261749889 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7f75d879-5e31-41fd-838c-b1f0408cb45a-metrics-certs") pod "network-metrics-daemon-vjfrn" (UID: "7f75d879-5e31-41fd-838c-b1f0408cb45a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 04:43:25.330273 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:25.330250    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 04:43:25.330273 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:25.330277    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 04:43:25.330439 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:25.330290    2579 projected.go:196] Error preparing data for projected volume kube-api-access-f8x7k for pod openshift-network-diagnostics/network-check-target-dgsdt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 04:43:25.330439 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:25.330339    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b1cae8a-12cf-4b59-85f0-a63f324fc8e1-kube-api-access-f8x7k podName:3b1cae8a-12cf-4b59-85f0-a63f324fc8e1 nodeName:}" failed. No retries permitted until 2026-10-30 04:43:29.33032336 +0000 UTC m=+10.262289084 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-f8x7k" (UniqueName: "kubernetes.io/projected/3b1cae8a-12cf-4b59-85f0-a63f324fc8e1-kube-api-access-f8x7k") pod "network-check-target-dgsdt" (UID: "3b1cae8a-12cf-4b59-85f0-a63f324fc8e1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 04:43:25.330439 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:25.330409    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 04:43:25.330439 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:25.330441    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41a4ec4b-b407-4316-9238-dee6be432d3c-original-pull-secret podName:41a4ec4b-b407-4316-9238-dee6be432d3c nodeName:}" failed. No retries permitted until 2026-10-30 04:43:29.330431153 +0000 UTC m=+10.262396862 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/41a4ec4b-b407-4316-9238-dee6be432d3c-original-pull-secret") pod "global-pull-secret-syncer-tt5mj" (UID: "41a4ec4b-b407-4316-9238-dee6be432d3c") : object "kube-system"/"original-pull-secret" not registered
Oct 30 04:43:25.554912 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:25.554871    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgsdt"
Oct 30 04:43:25.555081 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:25.554991    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgsdt" podUID="3b1cae8a-12cf-4b59-85f0-a63f324fc8e1"
Oct 30 04:43:25.555412 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:25.555393    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vjfrn"
Oct 30 04:43:25.555515 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:25.555473    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vjfrn" podUID="7f75d879-5e31-41fd-838c-b1f0408cb45a"
Oct 30 04:43:25.555578 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:25.555525    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tt5mj"
Oct 30 04:43:25.555638 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:25.555583    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tt5mj" podUID="41a4ec4b-b407-4316-9238-dee6be432d3c"
Oct 30 04:43:26.755448 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:26.755413    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6bpl4" event={"ID":"73a78a0d-d19e-4a96-b8ce-253d631da703","Type":"ContainerStarted","Data":"d98e2f4dead60898a7e103e2dc480a8b9385b8b220de4884d0fa68663e6f1dc3"}
Oct 30 04:43:26.795427 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:26.795368    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-6bpl4" podStartSLOduration=3.992052804 podStartE2EDuration="7.795348154s" totalImagesPullingTime="3.80329535s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:43:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:43:26.794866951 +0000 UTC m=+7.726832679" watchObservedRunningTime="2026-10-30 04:43:26.795348154 +0000 UTC m=+7.727313882"
Oct 30 04:43:27.552258 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:27.552197    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tt5mj"
Oct 30 04:43:27.552381 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:27.552318    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tt5mj" podUID="41a4ec4b-b407-4316-9238-dee6be432d3c"
Oct 30 04:43:27.552637 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:27.552618    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgsdt"
Oct 30 04:43:27.552710 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:27.552692    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgsdt" podUID="3b1cae8a-12cf-4b59-85f0-a63f324fc8e1"
Oct 30 04:43:27.552752 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:27.552743    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vjfrn"
Oct 30 04:43:27.552817 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:27.552804    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vjfrn" podUID="7f75d879-5e31-41fd-838c-b1f0408cb45a"
Oct 30 04:43:27.763226 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:27.763191    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhsc7" event={"ID":"672524df-2f52-455b-9423-a58f53c31129","Type":"ContainerStarted","Data":"5056a3e39a4ca836ba1440da89c883e270011873bd376cccbf308c64d6f648e8"}
Oct 30 04:43:27.766446 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:27.766419    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zqs6v" event={"ID":"daf6535c-7f33-407a-a5d9-e1c9525c8cb8","Type":"ContainerStarted","Data":"1491ae5db066d32f197471a844f5792fc6a7f886aa90f8cd6b91f1c7c098389d"}
Oct 30 04:43:28.989431 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:28.988895    2579 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 04:43:29.365175 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:29.365142    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/41a4ec4b-b407-4316-9238-dee6be432d3c-original-pull-secret\") pod \"global-pull-secret-syncer-tt5mj\" (UID: \"41a4ec4b-b407-4316-9238-dee6be432d3c\") " pod="kube-system/global-pull-secret-syncer-tt5mj"
Oct 30 04:43:29.365328 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:29.365183    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f75d879-5e31-41fd-838c-b1f0408cb45a-metrics-certs\") pod \"network-metrics-daemon-vjfrn\" (UID: \"7f75d879-5e31-41fd-838c-b1f0408cb45a\") " pod="openshift-multus/network-metrics-daemon-vjfrn"
Oct 30 04:43:29.365328 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:29.365223    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8x7k\" (UniqueName: \"kubernetes.io/projected/3b1cae8a-12cf-4b59-85f0-a63f324fc8e1-kube-api-access-f8x7k\") pod \"network-check-target-dgsdt\" (UID: \"3b1cae8a-12cf-4b59-85f0-a63f324fc8e1\") " pod="openshift-network-diagnostics/network-check-target-dgsdt"
Oct 30 04:43:29.365408 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:29.365330    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 04:43:29.365408 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:29.365345    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 04:43:29.365408 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:29.365355    2579 projected.go:196] Error preparing data for projected volume kube-api-access-f8x7k for pod openshift-network-diagnostics/network-check-target-dgsdt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 04:43:29.365408 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:29.365395    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b1cae8a-12cf-4b59-85f0-a63f324fc8e1-kube-api-access-f8x7k podName:3b1cae8a-12cf-4b59-85f0-a63f324fc8e1 nodeName:}" failed. No retries permitted until 2026-10-30 04:43:37.365384233 +0000 UTC m=+18.297349938 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-f8x7k" (UniqueName: "kubernetes.io/projected/3b1cae8a-12cf-4b59-85f0-a63f324fc8e1-kube-api-access-f8x7k") pod "network-check-target-dgsdt" (UID: "3b1cae8a-12cf-4b59-85f0-a63f324fc8e1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 04:43:29.365699 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:29.365685    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 04:43:29.365753 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:29.365726    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41a4ec4b-b407-4316-9238-dee6be432d3c-original-pull-secret podName:41a4ec4b-b407-4316-9238-dee6be432d3c nodeName:}" failed. No retries permitted until 2026-10-30 04:43:37.365715984 +0000 UTC m=+18.297681689 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/41a4ec4b-b407-4316-9238-dee6be432d3c-original-pull-secret") pod "global-pull-secret-syncer-tt5mj" (UID: "41a4ec4b-b407-4316-9238-dee6be432d3c") : object "kube-system"/"original-pull-secret" not registered
Oct 30 04:43:29.365792 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:29.365760    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 04:43:29.365792 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:29.365778    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f75d879-5e31-41fd-838c-b1f0408cb45a-metrics-certs podName:7f75d879-5e31-41fd-838c-b1f0408cb45a nodeName:}" failed. No retries permitted until 2026-10-30 04:43:37.365772503 +0000 UTC m=+18.297738208 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7f75d879-5e31-41fd-838c-b1f0408cb45a-metrics-certs") pod "network-metrics-daemon-vjfrn" (UID: "7f75d879-5e31-41fd-838c-b1f0408cb45a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 04:43:29.543752 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:29.543658    2579 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T04:43:28.989401431Z","UUID":"729af570-3af9-47b8-b6c0-9293ee8a363a","Handler":null,"Name":"","Endpoint":""}
Oct 30 04:43:29.549787 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:29.549770    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 04:43:29.549868 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:29.549797    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 04:43:29.552229 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:29.552207    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgsdt"
Oct 30 04:43:29.552324 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:29.552217    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vjfrn"
Oct 30 04:43:29.552377 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:29.552332    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgsdt" podUID="3b1cae8a-12cf-4b59-85f0-a63f324fc8e1"
Oct 30 04:43:29.552377 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:29.552345    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tt5mj"
Oct 30 04:43:29.552472 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:29.552457    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vjfrn" podUID="7f75d879-5e31-41fd-838c-b1f0408cb45a"
Oct 30 04:43:29.552567 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:29.552548    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tt5mj" podUID="41a4ec4b-b407-4316-9238-dee6be432d3c"
Oct 30 04:43:29.772496 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:29.772467    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gndxs" event={"ID":"885d476c-7568-4b63-ab9d-1da7608706f5","Type":"ContainerStarted","Data":"97d8e27e81c99a949ce43c5138c803210f1fefb3373729c35174d9feabdb833f"}
Oct 30 04:43:29.775473 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:29.775439    2579 generic.go:356] "Generic (PLEG): container finished" podID="a2a3eb4b-a128-4649-bf77-78df2705f147" containerID="70b5a7864be8534112c788ac29f42b3f3974399b71f6b90cd480b7b65466eb8f" exitCode=0
Oct 30 04:43:29.775562 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:29.775527    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tnwc6" event={"ID":"a2a3eb4b-a128-4649-bf77-78df2705f147","Type":"ContainerDied","Data":"70b5a7864be8534112c788ac29f42b3f3974399b71f6b90cd480b7b65466eb8f"}
Oct 30 04:43:29.779608 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:29.779491    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhsc7" event={"ID":"672524df-2f52-455b-9423-a58f53c31129","Type":"ContainerStarted","Data":"d40367283e01486a2683b94450c324af434831f009f52877d26f1a0ecd7870c5"}
Oct 30 04:43:29.781191 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:29.780806    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-c9tpc" event={"ID":"bf49335f-7b8e-4212-8777-1575cd0929f1","Type":"ContainerStarted","Data":"c800a9da13d81fc4088646eaacb100fbc6cc77b326c8485f387b2632460cdbd2"}
Oct 30 04:43:29.814658 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:29.814430    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-zqs6v" podStartSLOduration=5.766810139 podStartE2EDuration="10.814418327s" totalImagesPullingTime="5.047608188s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:43:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:43:27.805984044 +0000 UTC m=+8.737949773" watchObservedRunningTime="2026-10-30 04:43:29.814418327 +0000 UTC m=+10.746384053"
Oct 30 04:43:29.814919 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:29.814794    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-gndxs" podStartSLOduration=3.211015821 podStartE2EDuration="10.814784114s" totalImagesPullingTime="7.603768293s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:43:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:43:29.814348911 +0000 UTC m=+10.746314638" watchObservedRunningTime="2026-10-30 04:43:29.814784114 +0000 UTC m=+10.746749843"
Oct 30 04:43:29.930422 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:29.929708    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-c9tpc" podStartSLOduration=3.32807551 podStartE2EDuration="10.929692316s" totalImagesPullingTime="7.601616806s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:43:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:43:29.858297581 +0000 UTC m=+10.790263327" watchObservedRunningTime="2026-10-30 04:43:29.929692316 +0000 UTC m=+10.861658045"
Oct 30 04:43:30.785977 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:30.785657    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhsc7" event={"ID":"672524df-2f52-455b-9423-a58f53c31129","Type":"ContainerStarted","Data":"009f22d27fb7e0781bebcca288b1462c5714887031114fdcaacf44b6197cdb86"}
Oct 30 04:43:30.788389 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:30.788357    2579 generic.go:356] "Generic (PLEG): container finished" podID="a2a3eb4b-a128-4649-bf77-78df2705f147" containerID="a8f7f10b87ebe77015fdc121beab034f6837531fdcb02d7513d08c0581b11105" exitCode=0
Oct 30 04:43:30.789323 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:30.789304    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tnwc6" event={"ID":"a2a3eb4b-a128-4649-bf77-78df2705f147","Type":"ContainerDied","Data":"a8f7f10b87ebe77015fdc121beab034f6837531fdcb02d7513d08c0581b11105"}
Oct 30 04:43:30.825251 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:30.825191    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhsc7" podStartSLOduration=4.1050668869999996 podStartE2EDuration="11.825176727s" totalImagesPullingTime="7.72010984s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:43:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:43:30.825015335 +0000 UTC m=+11.756981053" watchObservedRunningTime="2026-10-30 04:43:30.825176727 +0000 UTC m=+11.757142456"
Oct 30 04:43:31.553026 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:31.552957    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgsdt"
Oct 30 04:43:31.553026 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:31.552975    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vjfrn"
Oct 30 04:43:31.553202 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:31.553078    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgsdt" podUID="3b1cae8a-12cf-4b59-85f0-a63f324fc8e1"
Oct 30 04:43:31.553202 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:31.553131    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tt5mj"
Oct 30 04:43:31.553316 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:31.553232    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tt5mj" podUID="41a4ec4b-b407-4316-9238-dee6be432d3c"
Oct 30 04:43:31.553369 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:31.553325    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vjfrn" podUID="7f75d879-5e31-41fd-838c-b1f0408cb45a"
Oct 30 04:43:31.571767 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:31.571742    2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-6bpl4"
Oct 30 04:43:31.572292 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:31.572268    2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-6bpl4"
Oct 30 04:43:31.792322 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:31.792287    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-st55p" event={"ID":"bff48d36-3909-4838-b499-66e5081bcc90","Type":"ContainerStarted","Data":"2d789b9c426e94ef33aa6252a62e69d575b51e4b0be35f7385370c7c3e0fa863"}
Oct 30 04:43:31.795094 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:31.795070    2579 generic.go:356] "Generic (PLEG): container finished" podID="a2a3eb4b-a128-4649-bf77-78df2705f147" containerID="789be87cb8f6d7efccf0b242d6e5e83afbedda95f133412978302941a7957e30" exitCode=0
Oct 30 04:43:31.795276 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:31.795133    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tnwc6" event={"ID":"a2a3eb4b-a128-4649-bf77-78df2705f147","Type":"ContainerDied","Data":"789be87cb8f6d7efccf0b242d6e5e83afbedda95f133412978302941a7957e30"}
Oct 30 04:43:31.830314 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:31.830235    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-st55p" podStartSLOduration=4.949900906 podStartE2EDuration="12.83022286s" totalImagesPullingTime="7.880321954s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:43:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:43:31.82987565 +0000 UTC m=+12.761841376" watchObservedRunningTime="2026-10-30 04:43:31.83022286 +0000 UTC m=+12.762188588"
Oct 30 04:43:32.799452 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:32.799411    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qtb8v" event={"ID":"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935","Type":"ContainerStarted","Data":"e81913b2f91a6a66bf5894247d7d8ad7873331dec929acc66d6461981a4ae873"}
Oct 30 04:43:32.842072 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:32.842023    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-qtb8v" podStartSLOduration=3.718975067 podStartE2EDuration="13.842006225s" totalImagesPullingTime="10.123031158s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:43:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:43:32.841663564 +0000 UTC m=+13.773629290" watchObservedRunningTime="2026-10-30 04:43:32.842006225 +0000 UTC m=+13.773971961"
Oct 30 04:43:33.556152 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:33.555701    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tt5mj"
Oct 30 04:43:33.556152 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:33.555809    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgsdt"
Oct 30 04:43:33.556152 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:33.555826    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tt5mj" podUID="41a4ec4b-b407-4316-9238-dee6be432d3c"
Oct 30 04:43:33.556152 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:33.555841    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vjfrn"
Oct 30 04:43:33.556152 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:33.555976    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgsdt" podUID="3b1cae8a-12cf-4b59-85f0-a63f324fc8e1"
Oct 30 04:43:33.556152 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:33.556099    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vjfrn" podUID="7f75d879-5e31-41fd-838c-b1f0408cb45a"
Oct 30 04:43:33.805581 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:33.805476    2579 generic.go:356] "Generic (PLEG): container finished" podID="a2a3eb4b-a128-4649-bf77-78df2705f147" containerID="1dfcf90a6c53a224c3570c4940ae9c61cdb35c9d06f67dc06d05ee22f81ab23c" exitCode=0
Oct 30 04:43:33.806128 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:33.805976    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tnwc6" event={"ID":"a2a3eb4b-a128-4649-bf77-78df2705f147","Type":"ContainerDied","Data":"1dfcf90a6c53a224c3570c4940ae9c61cdb35c9d06f67dc06d05ee22f81ab23c"}
Oct 30 04:43:34.289872 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:34.289799    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-6bpl4"
Oct 30 04:43:34.290024 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:34.289957    2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 04:43:34.290874 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:34.290852    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6bpl4"
Oct 30 04:43:34.811319 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:34.811280    2579 generic.go:356] "Generic (PLEG): container finished" podID="a2a3eb4b-a128-4649-bf77-78df2705f147" containerID="93da0d17151a83471e41b64d2d9cff19580aeaa3e200eb2a4203f5308ceeb9cf" exitCode=0
Oct 30 04:43:34.811731 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:34.811360    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tnwc6" event={"ID":"a2a3eb4b-a128-4649-bf77-78df2705f147","Type":"ContainerDied","Data":"93da0d17151a83471e41b64d2d9cff19580aeaa3e200eb2a4203f5308ceeb9cf"}
Oct 30 04:43:35.552625 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:35.552589    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgsdt"
Oct 30 04:43:35.552792 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:35.552631    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tt5mj"
Oct 30 04:43:35.552792 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:35.552710    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgsdt" podUID="3b1cae8a-12cf-4b59-85f0-a63f324fc8e1"
Oct 30 04:43:35.552918 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:35.552797    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vjfrn"
Oct 30 04:43:35.552918 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:35.552877    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vjfrn" podUID="7f75d879-5e31-41fd-838c-b1f0408cb45a"
Oct 30 04:43:35.552982 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:35.552953    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tt5mj" podUID="41a4ec4b-b407-4316-9238-dee6be432d3c"
Oct 30 04:43:35.816639 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:35.816562    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tnwc6" event={"ID":"a2a3eb4b-a128-4649-bf77-78df2705f147","Type":"ContainerStarted","Data":"23afeab3a1ea34277db1856066dee25107fecb12449fec9129b26ad603f93eab"}
Oct 30 04:43:35.887034 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:35.885099    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-tnwc6" podStartSLOduration=7.09156987 podStartE2EDuration="16.885082317s" totalImagesPullingTime="9.793512447s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:43:19 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:43:35.884837178 +0000 UTC m=+16.816802905" watchObservedRunningTime="2026-10-30 04:43:35.885082317 +0000 UTC m=+16.817048050"
Oct 30 04:43:36.822210 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:36.822000    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv" event={"ID":"70169887-cc72-4fb5-96b5-e09e260d7a8b","Type":"ContainerStarted","Data":"95442cdcd2079900c05b26c565fdf9bbb35142d5717c4ade7c3b75e68532100d"}
Oct 30 04:43:36.822519 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:36.822298    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv" event={"ID":"70169887-cc72-4fb5-96b5-e09e260d7a8b","Type":"ContainerStarted","Data":"c11e510fe5f18ce13ca49341f5da6b9c3254e5e6f4f2164ab2b1f1432ace001a"}
Oct 30 04:43:36.822519 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:36.822345    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv" event={"ID":"70169887-cc72-4fb5-96b5-e09e260d7a8b","Type":"ContainerStarted","Data":"543d3e0c30b6745393ddc2841ea6d2a7b46e023c9d2aeb44e91047a58942dbe2"}
Oct 30 04:43:36.822519 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:36.822398    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv" event={"ID":"70169887-cc72-4fb5-96b5-e09e260d7a8b","Type":"ContainerStarted","Data":"b8b984b9c3452134e50faf57416c1746e5c42ca52558b183b91972878d07aa3e"}
Oct 30 04:43:36.822519 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:36.822441    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv" event={"ID":"70169887-cc72-4fb5-96b5-e09e260d7a8b","Type":"ContainerStarted","Data":"4dbf594a7ce739bc9ad7585be5aaf7299a4cf7d44ab92a80b65940efedc95aa9"}
Oct 30 04:43:36.822519 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:36.822492    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv" event={"ID":"70169887-cc72-4fb5-96b5-e09e260d7a8b","Type":"ContainerStarted","Data":"7864b1c1e1ef51ebc8dfaab6b05e8945f0e5ba9d95b2d24073bb349174cfafe2"}
Oct 30 04:43:37.426913 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:37.426876    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8x7k\" (UniqueName: \"kubernetes.io/projected/3b1cae8a-12cf-4b59-85f0-a63f324fc8e1-kube-api-access-f8x7k\") pod \"network-check-target-dgsdt\" (UID: \"3b1cae8a-12cf-4b59-85f0-a63f324fc8e1\") " pod="openshift-network-diagnostics/network-check-target-dgsdt"
Oct 30 04:43:37.427077 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:37.426921    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/41a4ec4b-b407-4316-9238-dee6be432d3c-original-pull-secret\") pod \"global-pull-secret-syncer-tt5mj\" (UID: \"41a4ec4b-b407-4316-9238-dee6be432d3c\") " pod="kube-system/global-pull-secret-syncer-tt5mj"
Oct 30 04:43:37.427077 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:37.426944    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f75d879-5e31-41fd-838c-b1f0408cb45a-metrics-certs\") pod \"network-metrics-daemon-vjfrn\" (UID: \"7f75d879-5e31-41fd-838c-b1f0408cb45a\") " pod="openshift-multus/network-metrics-daemon-vjfrn"
Oct 30 04:43:37.427077 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:37.427022    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 04:43:37.427077 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:37.427033    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 04:43:37.427077 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:37.427024    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 04:43:37.427077 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:37.427078    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41a4ec4b-b407-4316-9238-dee6be432d3c-original-pull-secret podName:41a4ec4b-b407-4316-9238-dee6be432d3c nodeName:}" failed. No retries permitted until 2026-10-30 04:43:53.427064556 +0000 UTC m=+34.359030261 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/41a4ec4b-b407-4316-9238-dee6be432d3c-original-pull-secret") pod "global-pull-secret-syncer-tt5mj" (UID: "41a4ec4b-b407-4316-9238-dee6be432d3c") : object "kube-system"/"original-pull-secret" not registered
Oct 30 04:43:37.427077 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:37.427081    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 04:43:37.427357 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:37.427092    2579 projected.go:196] Error preparing data for projected volume kube-api-access-f8x7k for pod openshift-network-diagnostics/network-check-target-dgsdt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 04:43:37.427357 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:37.427094    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f75d879-5e31-41fd-838c-b1f0408cb45a-metrics-certs podName:7f75d879-5e31-41fd-838c-b1f0408cb45a nodeName:}" failed. No retries permitted until 2026-10-30 04:43:53.427086362 +0000 UTC m=+34.359052067 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7f75d879-5e31-41fd-838c-b1f0408cb45a-metrics-certs") pod "network-metrics-daemon-vjfrn" (UID: "7f75d879-5e31-41fd-838c-b1f0408cb45a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 04:43:37.427357 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:37.427129    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b1cae8a-12cf-4b59-85f0-a63f324fc8e1-kube-api-access-f8x7k podName:3b1cae8a-12cf-4b59-85f0-a63f324fc8e1 nodeName:}" failed. No retries permitted until 2026-10-30 04:43:53.427115669 +0000 UTC m=+34.359081377 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-f8x7k" (UniqueName: "kubernetes.io/projected/3b1cae8a-12cf-4b59-85f0-a63f324fc8e1-kube-api-access-f8x7k") pod "network-check-target-dgsdt" (UID: "3b1cae8a-12cf-4b59-85f0-a63f324fc8e1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 04:43:37.444194 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:37.444170    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:43:37.555650 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:37.555626    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tt5mj"
Oct 30 04:43:37.555769 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:37.555730    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgsdt"
Oct 30 04:43:37.555769 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:37.555743    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tt5mj" podUID="41a4ec4b-b407-4316-9238-dee6be432d3c"
Oct 30 04:43:37.555845 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:37.555803    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgsdt" podUID="3b1cae8a-12cf-4b59-85f0-a63f324fc8e1"
Oct 30 04:43:37.555845 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:37.555835    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vjfrn"
Oct 30 04:43:37.555955 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:37.555937    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vjfrn" podUID="7f75d879-5e31-41fd-838c-b1f0408cb45a"
Oct 30 04:43:37.584290 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:37.584268    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:43:39.022681 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:39.022652    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:43:39.104843 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:39.104771    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:43:39.553064 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:39.552987    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tt5mj"
Oct 30 04:43:39.553189 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:39.553066    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tt5mj" podUID="41a4ec4b-b407-4316-9238-dee6be432d3c"
Oct 30 04:43:39.553189 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:39.553155    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgsdt"
Oct 30 04:43:39.553267 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:39.553229    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgsdt" podUID="3b1cae8a-12cf-4b59-85f0-a63f324fc8e1"
Oct 30 04:43:39.553303 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:39.553274    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vjfrn"
Oct 30 04:43:39.553344 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:39.553329    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vjfrn" podUID="7f75d879-5e31-41fd-838c-b1f0408cb45a"
Oct 30 04:43:39.831121 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:39.831049    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv" event={"ID":"70169887-cc72-4fb5-96b5-e09e260d7a8b","Type":"ContainerStarted","Data":"139beed67515dcbdf50a46fe9dba740e1288107867ac6fb37d76969da042d08a"}
Oct 30 04:43:41.552974 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:41.552768    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vjfrn"
Oct 30 04:43:41.553548 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:41.552799    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgsdt"
Oct 30 04:43:41.553548 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:41.553058    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vjfrn" podUID="7f75d879-5e31-41fd-838c-b1f0408cb45a"
Oct 30 04:43:41.553548 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:41.553159    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgsdt" podUID="3b1cae8a-12cf-4b59-85f0-a63f324fc8e1"
Oct 30 04:43:41.553548 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:41.552862    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tt5mj"
Oct 30 04:43:41.553548 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:41.553258    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tt5mj" podUID="41a4ec4b-b407-4316-9238-dee6be432d3c"
Oct 30 04:43:41.839147 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:41.839069    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv" event={"ID":"70169887-cc72-4fb5-96b5-e09e260d7a8b","Type":"ContainerStarted","Data":"d22d0484157a73a44f3c3ba6e1f347e168fdce06e5c1732460f815b5efe58aec"}
Oct 30 04:43:41.839303 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:41.839286    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:41.855694 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:41.855671    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:41.926878 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:41.926826    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv" podStartSLOduration=8.339262066 podStartE2EDuration="22.926813769s" totalImagesPullingTime="14.587551703s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:43:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:43:41.913248347 +0000 UTC m=+22.845214073" watchObservedRunningTime="2026-10-30 04:43:41.926813769 +0000 UTC m=+22.858779514"
Oct 30 04:43:42.244166 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:42.244074    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:43:42.329452 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:42.329420    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:43:42.759748 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:42.759718    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:43:42.841385 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:42.841347    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:42.841555 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:42.841420    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:42.857091 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:42.857065    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:43:42.867504 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:42.867464    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:43:42.969404 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:42.969322    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:43:43.028067 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:43.028040    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tt5mj"]
Oct 30 04:43:43.028188 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:43.028155    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tt5mj"
Oct 30 04:43:43.028245 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:43.028228    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tt5mj" podUID="41a4ec4b-b407-4316-9238-dee6be432d3c"
Oct 30 04:43:43.031555 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:43.031532    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dgsdt"]
Oct 30 04:43:43.031663 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:43.031612    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgsdt"
Oct 30 04:43:43.031712 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:43.031697    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgsdt" podUID="3b1cae8a-12cf-4b59-85f0-a63f324fc8e1"
Oct 30 04:43:43.032010 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:43.031991    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vjfrn"]
Oct 30 04:43:43.032086 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:43.032075    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vjfrn"
Oct 30 04:43:43.032161 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:43.032146    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vjfrn" podUID="7f75d879-5e31-41fd-838c-b1f0408cb45a"
Oct 30 04:43:43.072375 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:43.072350    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:43:43.187987 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:43.187954    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:43:43.289358 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:43.289298    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:43:43.389255 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:43.389224    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:43:43.472593 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:43.472564    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:43:43.776156 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:43.776085    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:43:43.880646 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:43.880618    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:43:44.081511 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:44.081438    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:43:44.552182 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:44.552109    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tt5mj"
Oct 30 04:43:44.552319 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:44.552208    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgsdt"
Oct 30 04:43:44.552319 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:44.552222    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tt5mj" podUID="41a4ec4b-b407-4316-9238-dee6be432d3c"
Oct 30 04:43:44.552319 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:44.552281    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgsdt" podUID="3b1cae8a-12cf-4b59-85f0-a63f324fc8e1"
Oct 30 04:43:44.552319 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:44.552296    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vjfrn"
Oct 30 04:43:44.552510 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:44.552371    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vjfrn" podUID="7f75d879-5e31-41fd-838c-b1f0408cb45a"
Oct 30 04:43:45.699295 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:45.699266    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:43:45.733054 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:45.732997    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gndxs_885d476c-7568-4b63-ab9d-1da7608706f5/dns-node-resolver/0.log"
Oct 30 04:43:45.945706 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:45.945682    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:43:46.036053 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:46.035966    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:43:46.119641 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:46.119612    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:43:46.406954 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:46.406815    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:43:46.552729 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:46.552704    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vjfrn"
Oct 30 04:43:46.552729 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:46.552719    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tt5mj"
Oct 30 04:43:46.553080 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:46.552706    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgsdt"
Oct 30 04:43:46.553080 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:46.552813    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vjfrn" podUID="7f75d879-5e31-41fd-838c-b1f0408cb45a"
Oct 30 04:43:46.553080 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:46.552861    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgsdt" podUID="3b1cae8a-12cf-4b59-85f0-a63f324fc8e1"
Oct 30 04:43:46.553080 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:46.552945    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tt5mj" podUID="41a4ec4b-b407-4316-9238-dee6be432d3c"
Oct 30 04:43:46.862161 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:46.862089    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:43:46.962815 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:46.962789    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:43:47.133503 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:47.133444    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zqs6v_daf6535c-7f33-407a-a5d9-e1c9525c8cb8/node-ca/0.log"
Oct 30 04:43:47.284130 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:47.284099    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:43:47.485099 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:47.485013    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:43:47.683968 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:47.683921    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:43:47.784629 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:47.784543    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:43:48.092836 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:48.092738    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:43:48.552942 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:48.552847    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vjfrn"
Oct 30 04:43:48.552942 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:48.552888    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tt5mj"
Oct 30 04:43:48.552942 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:48.552853    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgsdt"
Oct 30 04:43:48.553205 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:48.552980    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vjfrn" podUID="7f75d879-5e31-41fd-838c-b1f0408cb45a"
Oct 30 04:43:48.553205 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:48.553079    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tt5mj" podUID="41a4ec4b-b407-4316-9238-dee6be432d3c"
Oct 30 04:43:48.553205 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:48.553161    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgsdt" podUID="3b1cae8a-12cf-4b59-85f0-a63f324fc8e1"
Oct 30 04:43:48.895848 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:48.895746    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:43:49.070010 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:49.069982    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:43:49.294144 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:49.294074    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:43:49.479675 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:49.479642    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:43:49.688217 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:49.688152    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:43:50.087296 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:50.087222    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:43:50.187749 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:50.187718    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:43:50.552704 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:50.552627    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vjfrn"
Oct 30 04:43:50.552704 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:50.552627    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgsdt"
Oct 30 04:43:50.553232 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:50.552786    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vjfrn" podUID="7f75d879-5e31-41fd-838c-b1f0408cb45a"
Oct 30 04:43:50.553232 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:50.552828    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tt5mj"
Oct 30 04:43:50.553232 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:50.552891    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgsdt" podUID="3b1cae8a-12cf-4b59-85f0-a63f324fc8e1"
Oct 30 04:43:50.553232 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:50.552964    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tt5mj" podUID="41a4ec4b-b407-4316-9238-dee6be432d3c"
Oct 30 04:43:50.891495 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:50.891426    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:43:51.084794 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:51.084763    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:43:51.386563 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:51.386496    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:43:51.483319 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:51.483288    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:43:51.636302 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:51.636271    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:43:51.756363 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:51.756295    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:43:51.856728 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:51.856704    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:43:52.288090 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:52.288062    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:43:52.553113 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:52.553038    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vjfrn"
Oct 30 04:43:52.553113 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:52.553047    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgsdt"
Oct 30 04:43:52.553113 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:52.553078    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tt5mj"
Oct 30 04:43:52.553460 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:52.553211    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vjfrn" podUID="7f75d879-5e31-41fd-838c-b1f0408cb45a"
Oct 30 04:43:52.553460 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:52.553338    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgsdt" podUID="3b1cae8a-12cf-4b59-85f0-a63f324fc8e1"
Oct 30 04:43:52.553460 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:52.553407    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tt5mj" podUID="41a4ec4b-b407-4316-9238-dee6be432d3c"
Oct 30 04:43:52.883686 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:52.883617    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:43:53.080621 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:53.080590    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:43:53.444830 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:53.444801    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8x7k\" (UniqueName: \"kubernetes.io/projected/3b1cae8a-12cf-4b59-85f0-a63f324fc8e1-kube-api-access-f8x7k\") pod \"network-check-target-dgsdt\" (UID: \"3b1cae8a-12cf-4b59-85f0-a63f324fc8e1\") " pod="openshift-network-diagnostics/network-check-target-dgsdt"
Oct 30 04:43:53.444997 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:53.444838    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/41a4ec4b-b407-4316-9238-dee6be432d3c-original-pull-secret\") pod \"global-pull-secret-syncer-tt5mj\" (UID: \"41a4ec4b-b407-4316-9238-dee6be432d3c\") " pod="kube-system/global-pull-secret-syncer-tt5mj"
Oct 30 04:43:53.444997 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:53.444857    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f75d879-5e31-41fd-838c-b1f0408cb45a-metrics-certs\") pod \"network-metrics-daemon-vjfrn\" (UID: \"7f75d879-5e31-41fd-838c-b1f0408cb45a\") " pod="openshift-multus/network-metrics-daemon-vjfrn"
Oct 30 04:43:53.444997 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:53.444959    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 04:43:53.444997 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:53.444961    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 04:43:53.444997 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:53.444973    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 04:43:53.444997 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:53.444989    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 04:43:53.444997 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:53.445001    2579 projected.go:196] Error preparing data for projected volume kube-api-access-f8x7k for pod openshift-network-diagnostics/network-check-target-dgsdt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 04:43:53.445203 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:53.445004    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f75d879-5e31-41fd-838c-b1f0408cb45a-metrics-certs podName:7f75d879-5e31-41fd-838c-b1f0408cb45a nodeName:}" failed. No retries permitted until 2026-10-30 04:44:25.444991041 +0000 UTC m=+66.376956759 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7f75d879-5e31-41fd-838c-b1f0408cb45a-metrics-certs") pod "network-metrics-daemon-vjfrn" (UID: "7f75d879-5e31-41fd-838c-b1f0408cb45a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 04:43:53.445203 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:53.445047    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41a4ec4b-b407-4316-9238-dee6be432d3c-original-pull-secret podName:41a4ec4b-b407-4316-9238-dee6be432d3c nodeName:}" failed. No retries permitted until 2026-10-30 04:44:25.44503514 +0000 UTC m=+66.377000845 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/41a4ec4b-b407-4316-9238-dee6be432d3c-original-pull-secret") pod "global-pull-secret-syncer-tt5mj" (UID: "41a4ec4b-b407-4316-9238-dee6be432d3c") : object "kube-system"/"original-pull-secret" not registered
Oct 30 04:43:53.445203 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:53.445063    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b1cae8a-12cf-4b59-85f0-a63f324fc8e1-kube-api-access-f8x7k podName:3b1cae8a-12cf-4b59-85f0-a63f324fc8e1 nodeName:}" failed. No retries permitted until 2026-10-30 04:44:25.445053972 +0000 UTC m=+66.377019677 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-f8x7k" (UniqueName: "kubernetes.io/projected/3b1cae8a-12cf-4b59-85f0-a63f324fc8e1-kube-api-access-f8x7k") pod "network-check-target-dgsdt" (UID: "3b1cae8a-12cf-4b59-85f0-a63f324fc8e1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 04:43:53.482000 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:53.481973    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:43:53.686215 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:53.686184    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:43:53.794613 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:53.794551    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:43:54.083358 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:54.083288    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:43:54.552183 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:54.552116    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tt5mj"
Oct 30 04:43:54.552183 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:54.552134    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgsdt"
Oct 30 04:43:54.552379 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:54.552202    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vjfrn"
Oct 30 04:43:54.552379 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:54.552210    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tt5mj" podUID="41a4ec4b-b407-4316-9238-dee6be432d3c"
Oct 30 04:43:54.552379 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:54.552282    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgsdt" podUID="3b1cae8a-12cf-4b59-85f0-a63f324fc8e1"
Oct 30 04:43:54.552379 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:54.552346    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vjfrn" podUID="7f75d879-5e31-41fd-838c-b1f0408cb45a"
Oct 30 04:43:54.885464 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:54.885397    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:43:55.288866 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:55.288778    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:43:55.479130 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:55.479103    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:43:55.691806 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:55.691728    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:43:56.085229 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:56.085155    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:43:56.482947 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:56.482867    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:43:56.552748 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:56.552717    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgsdt"
Oct 30 04:43:56.552748 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:56.552748    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vjfrn"
Oct 30 04:43:56.552929 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:56.552748    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tt5mj"
Oct 30 04:43:56.552929 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:56.552815    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgsdt" podUID="3b1cae8a-12cf-4b59-85f0-a63f324fc8e1"
Oct 30 04:43:56.552929 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:56.552878    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vjfrn" podUID="7f75d879-5e31-41fd-838c-b1f0408cb45a"
Oct 30 04:43:56.553028 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:56.552944    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tt5mj" podUID="41a4ec4b-b407-4316-9238-dee6be432d3c"
Oct 30 04:43:56.690245 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:56.690207    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:43:57.236469 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:57.236440    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:43:57.339778 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:57.339749    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:43:58.552243 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:58.552208    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vjfrn"
Oct 30 04:43:58.552243 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:58.552239    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgsdt"
Oct 30 04:43:58.552770 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:58.552333    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vjfrn" podUID="7f75d879-5e31-41fd-838c-b1f0408cb45a"
Oct 30 04:43:58.552770 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:58.552394    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgsdt" podUID="3b1cae8a-12cf-4b59-85f0-a63f324fc8e1"
Oct 30 04:43:58.552770 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:43:58.552468    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tt5mj"
Oct 30 04:43:58.552770 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:43:58.552547    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tt5mj" podUID="41a4ec4b-b407-4316-9238-dee6be432d3c"
Oct 30 04:44:00.552285 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:00.552256    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vjfrn"
Oct 30 04:44:00.552697 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:00.552259    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tt5mj"
Oct 30 04:44:00.552697 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:44:00.552363    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vjfrn" podUID="7f75d879-5e31-41fd-838c-b1f0408cb45a"
Oct 30 04:44:00.552697 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:00.552380    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgsdt"
Oct 30 04:44:00.552697 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:44:00.552455    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tt5mj" podUID="41a4ec4b-b407-4316-9238-dee6be432d3c"
Oct 30 04:44:00.552697 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:44:00.552521    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgsdt" podUID="3b1cae8a-12cf-4b59-85f0-a63f324fc8e1"
Oct 30 04:44:00.780557 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:00.780525    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:00.878429 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:00.878398    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:00.974922 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:00.974631    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:01.075137 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:01.075106    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:01.175313 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:01.175228    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:01.276031 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:01.276004    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:02.264695 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:02.264664    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:02.419184 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:02.419150    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:02.513351 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:02.513322    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:02.552614 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:02.552567    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tt5mj"
Oct 30 04:44:02.552686 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:02.552646    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vjfrn"
Oct 30 04:44:02.552686 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:02.552669    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgsdt"
Oct 30 04:44:02.552786 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:44:02.552767    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vjfrn" podUID="7f75d879-5e31-41fd-838c-b1f0408cb45a"
Oct 30 04:44:02.552820 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:44:02.552800    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgsdt" podUID="3b1cae8a-12cf-4b59-85f0-a63f324fc8e1"
Oct 30 04:44:02.552879 ip-10-0-135-98 kubenswrapper[2579]: E1030 04:44:02.552864    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tt5mj" podUID="41a4ec4b-b407-4316-9238-dee6be432d3c"
Oct 30 04:44:02.983059 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:02.983026    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:03.086168 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:03.086135    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:03.401455 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:03.401429    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-98.ec2.internal" event="NodeReady"
Oct 30 04:44:03.401790 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:03.401554    2579 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 04:44:03.457944 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:03.457916    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jghzj"]
Oct 30 04:44:03.459797 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:03.459778    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jghzj"
Oct 30 04:44:03.461327 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:03.461306    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-srsjc"]
Oct 30 04:44:03.463168 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:03.463152    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-srsjc"
Oct 30 04:44:03.477647 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:03.477627    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jghzj"]
Oct 30 04:44:03.478178 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:03.478159    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-g782k\""
Oct 30 04:44:03.478331 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:03.478309    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 04:44:03.478423 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:03.478354    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 04:44:03.478423 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:03.478394    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 04:44:03.486659 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:03.486641    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 04:44:03.490753 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:03.490735    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-bbk54"]
Oct 30 04:44:03.492383 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:03.492368    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-srsjc"]
Oct 30 04:44:03.492461 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:03.492451    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bbk54"
Oct 30 04:44:03.508579 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:03.508560    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bbk54"]
Oct 30 04:44:03.577402 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:03.577377    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 04:44:03.577522 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:03.577457    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 04:44:03.577757 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:03.577743    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4jtzx\""
Oct 30 04:44:03.577813 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:03.577802    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 04:44:03.577842 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:03.577828    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 04:44:03.577869 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:03.577842    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 04:44:03.577869 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:03.577857    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-k8wtk\""
Oct 30 04:44:03.617954 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:03.617933    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ddad100f-6a08-4da8-a51a-f3f2fd5438bb-config-volume\") pod \"dns-default-bbk54\" (UID: \"ddad100f-6a08-4da8-a51a-f3f2fd5438bb\") " pod="openshift-dns/dns-default-bbk54"
Oct 30 04:44:03.618021 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:03.617959    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d-data-volume\") pod \"insights-runtime-extractor-srsjc\" (UID: \"d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d\") " pod="openshift-insights/insights-runtime-extractor-srsjc"
Oct 30 04:44:03.618021 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:03.617980    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddad100f-6a08-4da8-a51a-f3f2fd5438bb-tmp-dir\") pod \"dns-default-bbk54\" (UID: \"ddad100f-6a08-4da8-a51a-f3f2fd5438bb\") " pod="openshift-dns/dns-default-bbk54"
Oct 30 04:44:03.618021 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:03.618005    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmbs5\" (UniqueName: \"kubernetes.io/projected/ddad100f-6a08-4da8-a51a-f3f2fd5438bb-kube-api-access-vmbs5\") pod \"dns-default-bbk54\" (UID: \"ddad100f-6a08-4da8-a51a-f3f2fd5438bb\") " pod="openshift-dns/dns-default-bbk54"
Oct 30 04:44:03.618111 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:03.618084    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv4sg\" (UniqueName: \"kubernetes.io/projected/d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d-kube-api-access-gv4sg\") pod \"insights-runtime-extractor-srsjc\" (UID: \"d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d\") " pod="openshift-insights/insights-runtime-extractor-srsjc"
Oct 30 04:44:03.618150 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:03.618136    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ddad100f-6a08-4da8-a51a-f3f2fd5438bb-metrics-tls\") pod \"dns-default-bbk54\" (UID: \"ddad100f-6a08-4da8-a51a-f3f2fd5438bb\") " pod="openshift-dns/dns-default-bbk54"
Oct 30 04:44:03.618197 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:03.618186    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ad09519-e98b-4efc-8b6b-ec7d031c3c2f-cert\") pod \"ingress-canary-jghzj\" (UID: \"7ad09519-e98b-4efc-8b6b-ec7d031c3c2f\") " pod="openshift-ingress-canary/ingress-canary-jghzj"
Oct 30 04:44:03.618229 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:03.618207    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qld8j\" (UniqueName: \"kubernetes.io/projected/7ad09519-e98b-4efc-8b6b-ec7d031c3c2f-kube-api-access-qld8j\") pod \"ingress-canary-jghzj\" (UID: \"7ad09519-e98b-4efc-8b6b-ec7d031c3c2f\") " pod="openshift-ingress-canary/ingress-canary-jghzj"
Oct 30 04:44:03.618257 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:03.618227    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-srsjc\" (UID: \"d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d\") " pod="openshift-insights/insights-runtime-extractor-srsjc"
Oct 30 04:44:03.618257 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:03.618244    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d-crio-socket\") pod \"insights-runtime-extractor-srsjc\" (UID: \"d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d\") " pod="openshift-insights/insights-runtime-extractor-srsjc"
Oct 30 04:44:03.618316 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:03.618299    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-srsjc\" (UID: \"d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d\") " pod="openshift-insights/insights-runtime-extractor-srsjc"
Oct 30 04:44:03.718836 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:03.718783    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ad09519-e98b-4efc-8b6b-ec7d031c3c2f-cert\") pod \"ingress-canary-jghzj\" (UID: \"7ad09519-e98b-4efc-8b6b-ec7d031c3c2f\") " pod="openshift-ingress-canary/ingress-canary-jghzj"
Oct 30 04:44:03.718836 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:03.718811    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qld8j\" (UniqueName: \"kubernetes.io/projected/7ad09519-e98b-4efc-8b6b-ec7d031c3c2f-kube-api-access-qld8j\") pod \"ingress-canary-jghzj\" (UID: \"7ad09519-e98b-4efc-8b6b-ec7d031c3c2f\") " pod="openshift-ingress-canary/ingress-canary-jghzj"
Oct 30 04:44:03.718836 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:03.718828    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-srsjc\" (UID: \"d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d\") " pod="openshift-insights/insights-runtime-extractor-srsjc"
Oct 30 04:44:03.719041 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:03.719021    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d-crio-socket\") pod \"insights-runtime-extractor-srsjc\" (UID: \"d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d\") " pod="openshift-insights/insights-runtime-extractor-srsjc"
Oct 30 04:44:03.719077 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:03.719060    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-srsjc\" (UID: \"d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d\") " pod="openshift-insights/insights-runtime-extractor-srsjc"
Oct 30 04:44:03.719148 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:03.719133    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ddad100f-6a08-4da8-a51a-f3f2fd5438bb-config-volume\") pod \"dns-default-bbk54\" (UID: \"ddad100f-6a08-4da8-a51a-f3f2fd5438bb\") " pod="openshift-dns/dns-default-bbk54"
Oct 30 04:44:03.719189 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:03.719163    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d-data-volume\") pod \"insights-runtime-extractor-srsjc\" (UID: \"d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d\") " pod="openshift-insights/insights-runtime-extractor-srsjc"
Oct 30 04:44:03.719220 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:03.719192    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddad100f-6a08-4da8-a51a-f3f2fd5438bb-tmp-dir\") pod \"dns-default-bbk54\" (UID: \"ddad100f-6a08-4da8-a51a-f3f2fd5438bb\") " pod="openshift-dns/dns-default-bbk54"
Oct 30 04:44:03.719220 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:03.719215    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmbs5\" (UniqueName: \"kubernetes.io/projected/ddad100f-6a08-4da8-a51a-f3f2fd5438bb-kube-api-access-vmbs5\") pod \"dns-default-bbk54\" (UID: \"ddad100f-6a08-4da8-a51a-f3f2fd5438bb\") " pod="openshift-dns/dns-default-bbk54"
Oct 30 04:44:03.719282 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:03.719251    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d-crio-socket\") pod \"insights-runtime-extractor-srsjc\" (UID: \"d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d\") " pod="openshift-insights/insights-runtime-extractor-srsjc"
Oct 30 04:44:03.719282 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:03.719262    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv4sg\" (UniqueName: \"kubernetes.io/projected/d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d-kube-api-access-gv4sg\") pod \"insights-runtime-extractor-srsjc\" (UID: \"d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d\") " pod="openshift-insights/insights-runtime-extractor-srsjc"
Oct 30 04:44:03.719347 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:03.719307    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ddad100f-6a08-4da8-a51a-f3f2fd5438bb-metrics-tls\") pod \"dns-default-bbk54\" (UID: \"ddad100f-6a08-4da8-a51a-f3f2fd5438bb\") " pod="openshift-dns/dns-default-bbk54"
Oct 30 04:44:03.719729 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:03.719693    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddad100f-6a08-4da8-a51a-f3f2fd5438bb-tmp-dir\") pod \"dns-default-bbk54\" (UID: \"ddad100f-6a08-4da8-a51a-f3f2fd5438bb\") " pod="openshift-dns/dns-default-bbk54"
Oct 30 04:44:03.719825 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:03.719734    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-srsjc\" (UID: \"d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d\") " pod="openshift-insights/insights-runtime-extractor-srsjc"
Oct 30 04:44:03.719825 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:03.719740    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d-data-volume\") pod \"insights-runtime-extractor-srsjc\" (UID: \"d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d\") " pod="openshift-insights/insights-runtime-extractor-srsjc"
Oct 30 04:44:03.719825 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:03.719758    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ddad100f-6a08-4da8-a51a-f3f2fd5438bb-config-volume\") pod \"dns-default-bbk54\" (UID: \"ddad100f-6a08-4da8-a51a-f3f2fd5438bb\") " pod="openshift-dns/dns-default-bbk54"
Oct 30 04:44:03.722598 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:03.722570    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ddad100f-6a08-4da8-a51a-f3f2fd5438bb-metrics-tls\") pod \"dns-default-bbk54\" (UID: \"ddad100f-6a08-4da8-a51a-f3f2fd5438bb\") " pod="openshift-dns/dns-default-bbk54"
Oct 30 04:44:03.722677 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:03.722648    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-srsjc\" (UID: \"d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d\") " pod="openshift-insights/insights-runtime-extractor-srsjc"
Oct 30 04:44:03.722722 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:03.722703    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ad09519-e98b-4efc-8b6b-ec7d031c3c2f-cert\") pod \"ingress-canary-jghzj\" (UID: \"7ad09519-e98b-4efc-8b6b-ec7d031c3c2f\") " pod="openshift-ingress-canary/ingress-canary-jghzj"
Oct 30 04:44:03.739373 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:03.739345    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv4sg\" (UniqueName: \"kubernetes.io/projected/d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d-kube-api-access-gv4sg\") pod \"insights-runtime-extractor-srsjc\" (UID: \"d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d\") " pod="openshift-insights/insights-runtime-extractor-srsjc"
Oct 30 04:44:03.739446 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:03.739372    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmbs5\" (UniqueName: \"kubernetes.io/projected/ddad100f-6a08-4da8-a51a-f3f2fd5438bb-kube-api-access-vmbs5\") pod \"dns-default-bbk54\" (UID: \"ddad100f-6a08-4da8-a51a-f3f2fd5438bb\") " pod="openshift-dns/dns-default-bbk54"
Oct 30 04:44:03.740524 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:03.740501    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qld8j\" (UniqueName: \"kubernetes.io/projected/7ad09519-e98b-4efc-8b6b-ec7d031c3c2f-kube-api-access-qld8j\") pod \"ingress-canary-jghzj\" (UID: \"7ad09519-e98b-4efc-8b6b-ec7d031c3c2f\") " pod="openshift-ingress-canary/ingress-canary-jghzj"
Oct 30 04:44:03.771501 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:03.771465    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jghzj"
Oct 30 04:44:03.777970 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:03.777956    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-srsjc"
Oct 30 04:44:03.802685 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:03.802642    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bbk54"
Oct 30 04:44:04.078702 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:04.078676    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jghzj"]
Oct 30 04:44:04.080440 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:44:04.080413    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ad09519_e98b_4efc_8b6b_ec7d031c3c2f.slice/crio-efb8094f5548de960a06cbfadeed8eb0424196b2bcdd35525ef699ec6ff36aa4 WatchSource:0}: Error finding container efb8094f5548de960a06cbfadeed8eb0424196b2bcdd35525ef699ec6ff36aa4: Status 404 returned error can't find the container with id efb8094f5548de960a06cbfadeed8eb0424196b2bcdd35525ef699ec6ff36aa4
Oct 30 04:44:04.090767 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:04.090747    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-srsjc"]
Oct 30 04:44:04.114448 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:04.114429    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bbk54"]
Oct 30 04:44:04.121782 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:44:04.121758    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddad100f_6a08_4da8_a51a_f3f2fd5438bb.slice/crio-83f871ee138c5e31cdb59e3f27b898b80540b16fb8ac9bad0f2056868d7dc4f4 WatchSource:0}: Error finding container 83f871ee138c5e31cdb59e3f27b898b80540b16fb8ac9bad0f2056868d7dc4f4: Status 404 returned error can't find the container with id 83f871ee138c5e31cdb59e3f27b898b80540b16fb8ac9bad0f2056868d7dc4f4
Oct 30 04:44:04.239532 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:04.239505    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gndxs_885d476c-7568-4b63-ab9d-1da7608706f5/dns-node-resolver/0.log"
Oct 30 04:44:04.552444 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:04.552362    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tt5mj"
Oct 30 04:44:04.552883 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:04.552791    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgsdt"
Oct 30 04:44:04.553290 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:04.553216    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vjfrn"
Oct 30 04:44:04.569086 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:04.569013    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 04:44:04.569382 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:04.569321    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 04:44:04.569382 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:04.569352    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 04:44:04.569540 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:04.569422    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k257x\""
Oct 30 04:44:04.569540 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:04.569443    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mxnkb\""
Oct 30 04:44:04.569540 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:04.569482    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-56dd5\""
Oct 30 04:44:04.569540 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:04.569511    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 04:44:04.890447 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:04.890358    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bbk54" event={"ID":"ddad100f-6a08-4da8-a51a-f3f2fd5438bb","Type":"ContainerStarted","Data":"83f871ee138c5e31cdb59e3f27b898b80540b16fb8ac9bad0f2056868d7dc4f4"}
Oct 30 04:44:04.891847 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:04.891795    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jghzj" event={"ID":"7ad09519-e98b-4efc-8b6b-ec7d031c3c2f","Type":"ContainerStarted","Data":"efb8094f5548de960a06cbfadeed8eb0424196b2bcdd35525ef699ec6ff36aa4"}
Oct 30 04:44:04.894328 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:04.894237    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-srsjc" event={"ID":"d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d","Type":"ContainerStarted","Data":"a19d4b28b330f3959f555ea5d71823e6ce82b05b7f2d1df82a5ba2d87c69d564"}
Oct 30 04:44:04.894328 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:04.894284    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-srsjc" event={"ID":"d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d","Type":"ContainerStarted","Data":"ac080b00868cc1ebc11f9863f82982d75a84a530a9fa75379ba3eb30ba200e49"}
Oct 30 04:44:04.894328 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:04.894305    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-srsjc" event={"ID":"d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d","Type":"ContainerStarted","Data":"6653524753f250fb105e5601c4942691e41dc39723b6868b480553e6e4fd3d16"}
Oct 30 04:44:04.937874 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:04.937846    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:05.041190 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:05.041155    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:05.070467 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:05.070390    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-4z6wz"]
Oct 30 04:44:05.072586 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:05.072559    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-4z6wz"
Oct 30 04:44:05.089767 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:05.089739    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 30 04:44:05.089929 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:05.089886    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 04:44:05.091112 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:05.091086    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-4z6wz"]
Oct 30 04:44:05.091287 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:05.091270    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 30 04:44:05.091373 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:05.091357    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 04:44:05.091425 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:05.091413    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-fmr9k\""
Oct 30 04:44:05.091477 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:05.091438    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 04:44:05.230664 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:05.230634    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3023960e-7dab-48fe-926b-b016f2e2b5b7-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-4z6wz\" (UID: \"3023960e-7dab-48fe-926b-b016f2e2b5b7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4z6wz"
Oct 30 04:44:05.230851 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:05.230672    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3023960e-7dab-48fe-926b-b016f2e2b5b7-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-4z6wz\" (UID: \"3023960e-7dab-48fe-926b-b016f2e2b5b7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4z6wz"
Oct 30 04:44:05.230851 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:05.230712    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhgw5\" (UniqueName: \"kubernetes.io/projected/3023960e-7dab-48fe-926b-b016f2e2b5b7-kube-api-access-mhgw5\") pod \"prometheus-operator-9cd89b74c-4z6wz\" (UID: \"3023960e-7dab-48fe-926b-b016f2e2b5b7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4z6wz"
Oct 30 04:44:05.230851 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:05.230742    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3023960e-7dab-48fe-926b-b016f2e2b5b7-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-4z6wz\" (UID: \"3023960e-7dab-48fe-926b-b016f2e2b5b7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4z6wz"
Oct 30 04:44:05.250350 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:05.250316    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:05.332041 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:05.332005    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3023960e-7dab-48fe-926b-b016f2e2b5b7-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-4z6wz\" (UID: \"3023960e-7dab-48fe-926b-b016f2e2b5b7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4z6wz"
Oct 30 04:44:05.332199 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:05.332055    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3023960e-7dab-48fe-926b-b016f2e2b5b7-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-4z6wz\" (UID: \"3023960e-7dab-48fe-926b-b016f2e2b5b7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4z6wz"
Oct 30 04:44:05.332199 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:05.332097    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhgw5\" (UniqueName: \"kubernetes.io/projected/3023960e-7dab-48fe-926b-b016f2e2b5b7-kube-api-access-mhgw5\") pod \"prometheus-operator-9cd89b74c-4z6wz\" (UID: \"3023960e-7dab-48fe-926b-b016f2e2b5b7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4z6wz"
Oct 30 04:44:05.332199 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:05.332126    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3023960e-7dab-48fe-926b-b016f2e2b5b7-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-4z6wz\" (UID: \"3023960e-7dab-48fe-926b-b016f2e2b5b7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4z6wz"
Oct 30 04:44:05.332864 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:05.332842    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3023960e-7dab-48fe-926b-b016f2e2b5b7-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-4z6wz\" (UID: \"3023960e-7dab-48fe-926b-b016f2e2b5b7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4z6wz"
Oct 30 04:44:05.347995 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:05.347967    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3023960e-7dab-48fe-926b-b016f2e2b5b7-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-4z6wz\" (UID: \"3023960e-7dab-48fe-926b-b016f2e2b5b7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4z6wz"
Oct 30 04:44:05.349115 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:05.348485    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3023960e-7dab-48fe-926b-b016f2e2b5b7-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-4z6wz\" (UID: \"3023960e-7dab-48fe-926b-b016f2e2b5b7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4z6wz"
Oct 30 04:44:05.353462 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:05.353438    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhgw5\" (UniqueName: \"kubernetes.io/projected/3023960e-7dab-48fe-926b-b016f2e2b5b7-kube-api-access-mhgw5\") pod \"prometheus-operator-9cd89b74c-4z6wz\" (UID: \"3023960e-7dab-48fe-926b-b016f2e2b5b7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4z6wz"
Oct 30 04:44:05.386970 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:05.386945    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-4z6wz"
Oct 30 04:44:05.685954 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:05.685864    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-4z6wz"]
Oct 30 04:44:05.689607 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:44:05.689563    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3023960e_7dab_48fe_926b_b016f2e2b5b7.slice/crio-744019019c1faa264b370e28aa0390f184677b13e9690c64273d2b0a8e103943 WatchSource:0}: Error finding container 744019019c1faa264b370e28aa0390f184677b13e9690c64273d2b0a8e103943: Status 404 returned error can't find the container with id 744019019c1faa264b370e28aa0390f184677b13e9690c64273d2b0a8e103943
Oct 30 04:44:05.845267 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:05.845170    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zqs6v_daf6535c-7f33-407a-a5d9-e1c9525c8cb8/node-ca/0.log"
Oct 30 04:44:05.898215 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:05.898182    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-4z6wz" event={"ID":"3023960e-7dab-48fe-926b-b016f2e2b5b7","Type":"ContainerStarted","Data":"744019019c1faa264b370e28aa0390f184677b13e9690c64273d2b0a8e103943"}
Oct 30 04:44:06.904095 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:06.904059    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-srsjc" event={"ID":"d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d","Type":"ContainerStarted","Data":"0d1d3bc0b51520357d3b6ca324fd859c1b5d76947b022762fa9c52c4208eedba"}
Oct 30 04:44:06.906012 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:06.905986    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bbk54" event={"ID":"ddad100f-6a08-4da8-a51a-f3f2fd5438bb","Type":"ContainerStarted","Data":"2bd78633f27ab4736bbe09ba8097e1813709ae1d8661a89cf756553cf49b5664"}
Oct 30 04:44:06.906121 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:06.906039    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bbk54" event={"ID":"ddad100f-6a08-4da8-a51a-f3f2fd5438bb","Type":"ContainerStarted","Data":"c0d1e1b51ac5b640807d5bc82dcbbf0cf184a653e6cedc79c7c58d0895cd6af2"}
Oct 30 04:44:06.906121 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:06.906071    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-bbk54"
Oct 30 04:44:06.946017 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:06.945963    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-srsjc" podStartSLOduration=1.838507313 podStartE2EDuration="3.945949057s" totalImagesPullingTime="2.107441744s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:44:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:44:06.943825442 +0000 UTC m=+47.875791180" watchObservedRunningTime="2026-10-30 04:44:06.945949057 +0000 UTC m=+47.877914785"
Oct 30 04:44:06.983475 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:06.983414    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-bbk54" podStartSLOduration=2.189105034 podStartE2EDuration="3.983396854s" totalImagesPullingTime="1.79429182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:44:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:44:06.982973085 +0000 UTC m=+47.914938814" watchObservedRunningTime="2026-10-30 04:44:06.983396854 +0000 UTC m=+47.915362582"
Oct 30 04:44:07.910698 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:07.910659    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-4z6wz" event={"ID":"3023960e-7dab-48fe-926b-b016f2e2b5b7","Type":"ContainerStarted","Data":"1262058a18882ed985ea1140091e8e809694618d04f67d12997d2d43892db33c"}
Oct 30 04:44:07.911247 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:07.910718    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-4z6wz" event={"ID":"3023960e-7dab-48fe-926b-b016f2e2b5b7","Type":"ContainerStarted","Data":"66522e9555e3cf7a0546fad0990aa448aaeebe657553a4e2d7593fe924590c16"}
Oct 30 04:44:07.912094 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:07.912064    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jghzj" event={"ID":"7ad09519-e98b-4efc-8b6b-ec7d031c3c2f","Type":"ContainerStarted","Data":"b351010e3ac79052354238daa625e14e68ae37bcf75e00fdb3eb6db1d9567816"}
Oct 30 04:44:07.938703 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:07.938678    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:07.950753 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:07.950715    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-4z6wz" podStartSLOduration=1.501916838 podStartE2EDuration="2.95070425s" totalImagesPullingTime="1.448787412s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:44:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:44:07.949868447 +0000 UTC m=+48.881834174" watchObservedRunningTime="2026-10-30 04:44:07.95070425 +0000 UTC m=+48.882669989"
Oct 30 04:44:07.990380 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:07.990336    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jghzj" podStartSLOduration=2.220768399 podStartE2EDuration="4.990321576s" totalImagesPullingTime="2.769553177s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:44:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:44:07.989289182 +0000 UTC m=+48.921254908" watchObservedRunningTime="2026-10-30 04:44:07.990321576 +0000 UTC m=+48.922287304"
Oct 30 04:44:09.302260 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:09.302223    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:09.418442 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:09.418411    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:09.594445 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:09.594415    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:09.704889 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:09.704860    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9mr8m"]
Oct 30 04:44:09.705454 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:09.705421    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:09.707069 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:09.707054    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9mr8m"
Oct 30 04:44:09.738030 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:09.738005    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 04:44:09.738207 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:09.738193    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 04:44:09.738246 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:09.738209    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dzdcr\""
Oct 30 04:44:09.807829 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:09.807802    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 04:44:09.810289 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:09.810265    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:09.868427 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:09.868364    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/76081e78-ba20-4f7c-b825-a9e0da23c511-sys\") pod \"node-exporter-9mr8m\" (UID: \"76081e78-ba20-4f7c-b825-a9e0da23c511\") " pod="openshift-monitoring/node-exporter-9mr8m"
Oct 30 04:44:09.868427 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:09.868394    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/76081e78-ba20-4f7c-b825-a9e0da23c511-node-exporter-accelerators-collector-config\") pod \"node-exporter-9mr8m\" (UID: \"76081e78-ba20-4f7c-b825-a9e0da23c511\") " pod="openshift-monitoring/node-exporter-9mr8m"
Oct 30 04:44:09.868427 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:09.868418    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/76081e78-ba20-4f7c-b825-a9e0da23c511-node-exporter-tls\") pod \"node-exporter-9mr8m\" (UID: \"76081e78-ba20-4f7c-b825-a9e0da23c511\") " pod="openshift-monitoring/node-exporter-9mr8m"
Oct 30 04:44:09.868666 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:09.868489    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76081e78-ba20-4f7c-b825-a9e0da23c511-metrics-client-ca\") pod \"node-exporter-9mr8m\" (UID: \"76081e78-ba20-4f7c-b825-a9e0da23c511\") " pod="openshift-monitoring/node-exporter-9mr8m"
Oct 30 04:44:09.868666 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:09.868582    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/76081e78-ba20-4f7c-b825-a9e0da23c511-root\") pod \"node-exporter-9mr8m\" (UID: \"76081e78-ba20-4f7c-b825-a9e0da23c511\") " pod="openshift-monitoring/node-exporter-9mr8m"
Oct 30 04:44:09.868666 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:09.868613    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dg2h\" (UniqueName: \"kubernetes.io/projected/76081e78-ba20-4f7c-b825-a9e0da23c511-kube-api-access-4dg2h\") pod \"node-exporter-9mr8m\" (UID: \"76081e78-ba20-4f7c-b825-a9e0da23c511\") " pod="openshift-monitoring/node-exporter-9mr8m"
Oct 30 04:44:09.868666 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:09.868641    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/76081e78-ba20-4f7c-b825-a9e0da23c511-node-exporter-wtmp\") pod \"node-exporter-9mr8m\" (UID: \"76081e78-ba20-4f7c-b825-a9e0da23c511\") " pod="openshift-monitoring/node-exporter-9mr8m"
Oct 30 04:44:09.868794 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:09.868702    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/76081e78-ba20-4f7c-b825-a9e0da23c511-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9mr8m\" (UID: \"76081e78-ba20-4f7c-b825-a9e0da23c511\") " pod="openshift-monitoring/node-exporter-9mr8m"
Oct 30 04:44:09.868794 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:09.868721    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/76081e78-ba20-4f7c-b825-a9e0da23c511-node-exporter-textfile\") pod \"node-exporter-9mr8m\" (UID: \"76081e78-ba20-4f7c-b825-a9e0da23c511\") " pod="openshift-monitoring/node-exporter-9mr8m"
Oct 30 04:44:09.908959 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:09.908937    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:09.969269 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:09.969243    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/76081e78-ba20-4f7c-b825-a9e0da23c511-sys\") pod \"node-exporter-9mr8m\" (UID: \"76081e78-ba20-4f7c-b825-a9e0da23c511\") " pod="openshift-monitoring/node-exporter-9mr8m"
Oct 30 04:44:09.969447 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:09.969275    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/76081e78-ba20-4f7c-b825-a9e0da23c511-node-exporter-accelerators-collector-config\") pod \"node-exporter-9mr8m\" (UID: \"76081e78-ba20-4f7c-b825-a9e0da23c511\") " pod="openshift-monitoring/node-exporter-9mr8m"
Oct 30 04:44:09.969447 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:09.969296    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/76081e78-ba20-4f7c-b825-a9e0da23c511-node-exporter-tls\") pod \"node-exporter-9mr8m\" (UID: \"76081e78-ba20-4f7c-b825-a9e0da23c511\") " pod="openshift-monitoring/node-exporter-9mr8m"
Oct 30 04:44:09.969447 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:09.969322    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76081e78-ba20-4f7c-b825-a9e0da23c511-metrics-client-ca\") pod \"node-exporter-9mr8m\" (UID: \"76081e78-ba20-4f7c-b825-a9e0da23c511\") " pod="openshift-monitoring/node-exporter-9mr8m"
Oct 30 04:44:09.969447 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:09.969360    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/76081e78-ba20-4f7c-b825-a9e0da23c511-sys\") pod \"node-exporter-9mr8m\" (UID: \"76081e78-ba20-4f7c-b825-a9e0da23c511\") " pod="openshift-monitoring/node-exporter-9mr8m"
Oct 30 04:44:09.969640 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:09.969477    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/76081e78-ba20-4f7c-b825-a9e0da23c511-root\") pod \"node-exporter-9mr8m\" (UID: \"76081e78-ba20-4f7c-b825-a9e0da23c511\") " pod="openshift-monitoring/node-exporter-9mr8m"
Oct 30 04:44:09.969640 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:09.969506    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dg2h\" (UniqueName: \"kubernetes.io/projected/76081e78-ba20-4f7c-b825-a9e0da23c511-kube-api-access-4dg2h\") pod \"node-exporter-9mr8m\" (UID: \"76081e78-ba20-4f7c-b825-a9e0da23c511\") " pod="openshift-monitoring/node-exporter-9mr8m"
Oct 30 04:44:09.969640 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:09.969532    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/76081e78-ba20-4f7c-b825-a9e0da23c511-node-exporter-wtmp\") pod \"node-exporter-9mr8m\" (UID: \"76081e78-ba20-4f7c-b825-a9e0da23c511\") " pod="openshift-monitoring/node-exporter-9mr8m"
Oct 30 04:44:09.969640 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:09.969570    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/76081e78-ba20-4f7c-b825-a9e0da23c511-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9mr8m\" (UID: \"76081e78-ba20-4f7c-b825-a9e0da23c511\") " pod="openshift-monitoring/node-exporter-9mr8m"
Oct 30 04:44:09.969640 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:09.969582    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/76081e78-ba20-4f7c-b825-a9e0da23c511-root\") pod \"node-exporter-9mr8m\" (UID: \"76081e78-ba20-4f7c-b825-a9e0da23c511\") " pod="openshift-monitoring/node-exporter-9mr8m"
Oct 30 04:44:09.969640 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:09.969599    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/76081e78-ba20-4f7c-b825-a9e0da23c511-node-exporter-textfile\") pod \"node-exporter-9mr8m\" (UID: \"76081e78-ba20-4f7c-b825-a9e0da23c511\") " pod="openshift-monitoring/node-exporter-9mr8m"
Oct 30 04:44:09.969971 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:09.969709    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/76081e78-ba20-4f7c-b825-a9e0da23c511-node-exporter-wtmp\") pod \"node-exporter-9mr8m\" (UID: \"76081e78-ba20-4f7c-b825-a9e0da23c511\") " pod="openshift-monitoring/node-exporter-9mr8m"
Oct 30 04:44:09.969971 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:09.969897    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76081e78-ba20-4f7c-b825-a9e0da23c511-metrics-client-ca\") pod \"node-exporter-9mr8m\" (UID: \"76081e78-ba20-4f7c-b825-a9e0da23c511\") " pod="openshift-monitoring/node-exporter-9mr8m"
Oct 30 04:44:09.970115 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:09.970094    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/76081e78-ba20-4f7c-b825-a9e0da23c511-node-exporter-textfile\") pod \"node-exporter-9mr8m\" (UID: \"76081e78-ba20-4f7c-b825-a9e0da23c511\") " pod="openshift-monitoring/node-exporter-9mr8m"
Oct 30 04:44:09.970328 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:09.970305    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/76081e78-ba20-4f7c-b825-a9e0da23c511-node-exporter-accelerators-collector-config\") pod \"node-exporter-9mr8m\" (UID: \"76081e78-ba20-4f7c-b825-a9e0da23c511\") " pod="openshift-monitoring/node-exporter-9mr8m"
Oct 30 04:44:09.971598 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:09.971580    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/76081e78-ba20-4f7c-b825-a9e0da23c511-node-exporter-tls\") pod \"node-exporter-9mr8m\" (UID: \"76081e78-ba20-4f7c-b825-a9e0da23c511\") " pod="openshift-monitoring/node-exporter-9mr8m"
Oct 30 04:44:09.971691 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:09.971650    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/76081e78-ba20-4f7c-b825-a9e0da23c511-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9mr8m\" (UID: \"76081e78-ba20-4f7c-b825-a9e0da23c511\") " pod="openshift-monitoring/node-exporter-9mr8m"
Oct 30 04:44:09.990606 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:09.990578    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dg2h\" (UniqueName: \"kubernetes.io/projected/76081e78-ba20-4f7c-b825-a9e0da23c511-kube-api-access-4dg2h\") pod \"node-exporter-9mr8m\" (UID: \"76081e78-ba20-4f7c-b825-a9e0da23c511\") " pod="openshift-monitoring/node-exporter-9mr8m"
Oct 30 04:44:10.009006 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:10.008977    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:10.019920 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:10.019891    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9mr8m"
Oct 30 04:44:10.029094 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:44:10.029071    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76081e78_ba20_4f7c_b825_a9e0da23c511.slice/crio-cecccb7ad501a7ae51a7855e1a1426bcbdc1141471d05d904280a45ff1211b96 WatchSource:0}: Error finding container cecccb7ad501a7ae51a7855e1a1426bcbdc1141471d05d904280a45ff1211b96: Status 404 returned error can't find the container with id cecccb7ad501a7ae51a7855e1a1426bcbdc1141471d05d904280a45ff1211b96
Oct 30 04:44:10.214296 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:10.214212    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:10.846549 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:10.846519    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:10.921417 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:10.921394    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9mr8m" event={"ID":"76081e78-ba20-4f7c-b825-a9e0da23c511","Type":"ContainerStarted","Data":"cecccb7ad501a7ae51a7855e1a1426bcbdc1141471d05d904280a45ff1211b96"}
Oct 30 04:44:10.963201 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:10.963179    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:11.119119 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:11.119036    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:11.926373 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:11.926342    2579 generic.go:356] "Generic (PLEG): container finished" podID="76081e78-ba20-4f7c-b825-a9e0da23c511" containerID="512c27af9a0f400fe7957e95f53cd03c0306d65e5b9b599ca8e8e1fb62387359" exitCode=0
Oct 30 04:44:11.926758 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:11.926377    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9mr8m" event={"ID":"76081e78-ba20-4f7c-b825-a9e0da23c511","Type":"ContainerDied","Data":"512c27af9a0f400fe7957e95f53cd03c0306d65e5b9b599ca8e8e1fb62387359"}
Oct 30 04:44:11.955948 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:11.955918    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-95b457c5b-n92bg"]
Oct 30 04:44:11.960252 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:11.960232    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-95b457c5b-n92bg"
Oct 30 04:44:11.978355 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:11.978330    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 30 04:44:11.980097 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:11.980072    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-95b457c5b-n92bg"]
Oct 30 04:44:11.981247 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:11.981221    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 30 04:44:11.981247 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:11.981228    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-rdtp7\""
Oct 30 04:44:12.078772 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.078746    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 30 04:44:12.078772 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.078752    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 30 04:44:12.079003 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.078752    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 30 04:44:12.079003 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.078836    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-7hpknhdr5staj\""
Oct 30 04:44:12.085343 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.085320    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd380d55-276e-45eb-8191-c5940ed8385b-tmp\") pod \"thanos-querier-95b457c5b-n92bg\" (UID: \"fd380d55-276e-45eb-8191-c5940ed8385b\") " pod="openshift-monitoring/thanos-querier-95b457c5b-n92bg"
Oct 30 04:44:12.085449 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.085351    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fd380d55-276e-45eb-8191-c5940ed8385b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-95b457c5b-n92bg\" (UID: \"fd380d55-276e-45eb-8191-c5940ed8385b\") " pod="openshift-monitoring/thanos-querier-95b457c5b-n92bg"
Oct 30 04:44:12.085449 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.085385    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fd380d55-276e-45eb-8191-c5940ed8385b-secret-grpc-tls\") pod \"thanos-querier-95b457c5b-n92bg\" (UID: \"fd380d55-276e-45eb-8191-c5940ed8385b\") " pod="openshift-monitoring/thanos-querier-95b457c5b-n92bg"
Oct 30 04:44:12.085449 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.085412    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fd380d55-276e-45eb-8191-c5940ed8385b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-95b457c5b-n92bg\" (UID: \"fd380d55-276e-45eb-8191-c5940ed8385b\") " pod="openshift-monitoring/thanos-querier-95b457c5b-n92bg"
Oct 30 04:44:12.085618 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.085436    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fd380d55-276e-45eb-8191-c5940ed8385b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-95b457c5b-n92bg\" (UID: \"fd380d55-276e-45eb-8191-c5940ed8385b\") " pod="openshift-monitoring/thanos-querier-95b457c5b-n92bg"
Oct 30 04:44:12.085667 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.085646    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7tw7\" (UniqueName: \"kubernetes.io/projected/fd380d55-276e-45eb-8191-c5940ed8385b-kube-api-access-t7tw7\") pod \"thanos-querier-95b457c5b-n92bg\" (UID: \"fd380d55-276e-45eb-8191-c5940ed8385b\") " pod="openshift-monitoring/thanos-querier-95b457c5b-n92bg"
Oct 30 04:44:12.085719 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.085684    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd380d55-276e-45eb-8191-c5940ed8385b-metrics-client-ca\") pod \"thanos-querier-95b457c5b-n92bg\" (UID: \"fd380d55-276e-45eb-8191-c5940ed8385b\") " pod="openshift-monitoring/thanos-querier-95b457c5b-n92bg"
Oct 30 04:44:12.085769 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.085717    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fd380d55-276e-45eb-8191-c5940ed8385b-secret-thanos-querier-tls\") pod \"thanos-querier-95b457c5b-n92bg\" (UID: \"fd380d55-276e-45eb-8191-c5940ed8385b\") " pod="openshift-monitoring/thanos-querier-95b457c5b-n92bg"
Oct 30 04:44:12.085830 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.085782    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fd380d55-276e-45eb-8191-c5940ed8385b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-95b457c5b-n92bg\" (UID: \"fd380d55-276e-45eb-8191-c5940ed8385b\") " pod="openshift-monitoring/thanos-querier-95b457c5b-n92bg"
Oct 30 04:44:12.186229 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.186141    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7tw7\" (UniqueName: \"kubernetes.io/projected/fd380d55-276e-45eb-8191-c5940ed8385b-kube-api-access-t7tw7\") pod \"thanos-querier-95b457c5b-n92bg\" (UID: \"fd380d55-276e-45eb-8191-c5940ed8385b\") " pod="openshift-monitoring/thanos-querier-95b457c5b-n92bg"
Oct 30 04:44:12.186229 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.186195    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd380d55-276e-45eb-8191-c5940ed8385b-metrics-client-ca\") pod \"thanos-querier-95b457c5b-n92bg\" (UID: \"fd380d55-276e-45eb-8191-c5940ed8385b\") " pod="openshift-monitoring/thanos-querier-95b457c5b-n92bg"
Oct 30 04:44:12.186229 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.186222    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fd380d55-276e-45eb-8191-c5940ed8385b-secret-thanos-querier-tls\") pod \"thanos-querier-95b457c5b-n92bg\" (UID: \"fd380d55-276e-45eb-8191-c5940ed8385b\") " pod="openshift-monitoring/thanos-querier-95b457c5b-n92bg"
Oct 30 04:44:12.186482 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.186259    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fd380d55-276e-45eb-8191-c5940ed8385b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-95b457c5b-n92bg\" (UID: \"fd380d55-276e-45eb-8191-c5940ed8385b\") " pod="openshift-monitoring/thanos-querier-95b457c5b-n92bg"
Oct 30 04:44:12.186482 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.186391    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd380d55-276e-45eb-8191-c5940ed8385b-tmp\") pod \"thanos-querier-95b457c5b-n92bg\" (UID: \"fd380d55-276e-45eb-8191-c5940ed8385b\") " pod="openshift-monitoring/thanos-querier-95b457c5b-n92bg"
Oct 30 04:44:12.186482 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.186430    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fd380d55-276e-45eb-8191-c5940ed8385b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-95b457c5b-n92bg\" (UID: \"fd380d55-276e-45eb-8191-c5940ed8385b\") " pod="openshift-monitoring/thanos-querier-95b457c5b-n92bg"
Oct 30 04:44:12.186482 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.186453    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fd380d55-276e-45eb-8191-c5940ed8385b-secret-grpc-tls\") pod \"thanos-querier-95b457c5b-n92bg\" (UID: \"fd380d55-276e-45eb-8191-c5940ed8385b\") " pod="openshift-monitoring/thanos-querier-95b457c5b-n92bg"
Oct 30 04:44:12.186482 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.186477    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fd380d55-276e-45eb-8191-c5940ed8385b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-95b457c5b-n92bg\" (UID: \"fd380d55-276e-45eb-8191-c5940ed8385b\") " pod="openshift-monitoring/thanos-querier-95b457c5b-n92bg"
Oct 30 04:44:12.186721 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.186650    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fd380d55-276e-45eb-8191-c5940ed8385b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-95b457c5b-n92bg\" (UID: \"fd380d55-276e-45eb-8191-c5940ed8385b\") " pod="openshift-monitoring/thanos-querier-95b457c5b-n92bg"
Oct 30 04:44:12.187124 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.187101    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd380d55-276e-45eb-8191-c5940ed8385b-tmp\") pod \"thanos-querier-95b457c5b-n92bg\" (UID: \"fd380d55-276e-45eb-8191-c5940ed8385b\") " pod="openshift-monitoring/thanos-querier-95b457c5b-n92bg"
Oct 30 04:44:12.187227 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.187126    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd380d55-276e-45eb-8191-c5940ed8385b-metrics-client-ca\") pod \"thanos-querier-95b457c5b-n92bg\" (UID: \"fd380d55-276e-45eb-8191-c5940ed8385b\") " pod="openshift-monitoring/thanos-querier-95b457c5b-n92bg"
Oct 30 04:44:12.188954 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.188928    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fd380d55-276e-45eb-8191-c5940ed8385b-secret-thanos-querier-tls\") pod \"thanos-querier-95b457c5b-n92bg\" (UID: \"fd380d55-276e-45eb-8191-c5940ed8385b\") " pod="openshift-monitoring/thanos-querier-95b457c5b-n92bg"
Oct 30 04:44:12.189049 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.188931    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fd380d55-276e-45eb-8191-c5940ed8385b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-95b457c5b-n92bg\" (UID: \"fd380d55-276e-45eb-8191-c5940ed8385b\") " pod="openshift-monitoring/thanos-querier-95b457c5b-n92bg"
Oct 30 04:44:12.189294 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.189274    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fd380d55-276e-45eb-8191-c5940ed8385b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-95b457c5b-n92bg\" (UID: \"fd380d55-276e-45eb-8191-c5940ed8385b\") " pod="openshift-monitoring/thanos-querier-95b457c5b-n92bg"
Oct 30 04:44:12.189421 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.189400    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fd380d55-276e-45eb-8191-c5940ed8385b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-95b457c5b-n92bg\" (UID: \"fd380d55-276e-45eb-8191-c5940ed8385b\") " pod="openshift-monitoring/thanos-querier-95b457c5b-n92bg"
Oct 30 04:44:12.189458 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.189447    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fd380d55-276e-45eb-8191-c5940ed8385b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-95b457c5b-n92bg\" (UID: \"fd380d55-276e-45eb-8191-c5940ed8385b\") " pod="openshift-monitoring/thanos-querier-95b457c5b-n92bg"
Oct 30 04:44:12.189490 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.189455    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fd380d55-276e-45eb-8191-c5940ed8385b-secret-grpc-tls\") pod \"thanos-querier-95b457c5b-n92bg\" (UID: \"fd380d55-276e-45eb-8191-c5940ed8385b\") " pod="openshift-monitoring/thanos-querier-95b457c5b-n92bg"
Oct 30 04:44:12.205937 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.205892    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7tw7\" (UniqueName: \"kubernetes.io/projected/fd380d55-276e-45eb-8191-c5940ed8385b-kube-api-access-t7tw7\") pod \"thanos-querier-95b457c5b-n92bg\" (UID: \"fd380d55-276e-45eb-8191-c5940ed8385b\") " pod="openshift-monitoring/thanos-querier-95b457c5b-n92bg"
Oct 30 04:44:12.245704 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.245676    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:12.286302 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.286279    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-95b457c5b-n92bg"
Oct 30 04:44:12.346497 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.346469    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:12.601337 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.601293    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-95b457c5b-n92bg"]
Oct 30 04:44:12.643011 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.642990    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-f66c46c7c-skmxs"]
Oct 30 04:44:12.647040 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.647023    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f66c46c7c-skmxs"
Oct 30 04:44:12.662894 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.662872    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 30 04:44:12.663006 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.662892    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 30 04:44:12.663006 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.662967    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 04:44:12.663006 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.662999    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 30 04:44:12.663115 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.663026    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 30 04:44:12.663181 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.663163    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-j5jgn\""
Oct 30 04:44:12.663234 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.663178    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 30 04:44:12.663281 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.663265    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 04:44:12.664200 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.664178    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f66c46c7c-skmxs"]
Oct 30 04:44:12.791332 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.791298    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5q8q\" (UniqueName: \"kubernetes.io/projected/faa6a40d-2e9e-4348-a599-0df6f2d0177d-kube-api-access-p5q8q\") pod \"console-f66c46c7c-skmxs\" (UID: \"faa6a40d-2e9e-4348-a599-0df6f2d0177d\") " pod="openshift-console/console-f66c46c7c-skmxs"
Oct 30 04:44:12.791332 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.791337    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/faa6a40d-2e9e-4348-a599-0df6f2d0177d-console-serving-cert\") pod \"console-f66c46c7c-skmxs\" (UID: \"faa6a40d-2e9e-4348-a599-0df6f2d0177d\") " pod="openshift-console/console-f66c46c7c-skmxs"
Oct 30 04:44:12.791580 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.791353    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/faa6a40d-2e9e-4348-a599-0df6f2d0177d-console-config\") pod \"console-f66c46c7c-skmxs\" (UID: \"faa6a40d-2e9e-4348-a599-0df6f2d0177d\") " pod="openshift-console/console-f66c46c7c-skmxs"
Oct 30 04:44:12.791580 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.791432    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/faa6a40d-2e9e-4348-a599-0df6f2d0177d-console-oauth-config\") pod \"console-f66c46c7c-skmxs\" (UID: \"faa6a40d-2e9e-4348-a599-0df6f2d0177d\") " pod="openshift-console/console-f66c46c7c-skmxs"
Oct 30 04:44:12.791580 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.791509    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/faa6a40d-2e9e-4348-a599-0df6f2d0177d-tmp\") pod \"console-f66c46c7c-skmxs\" (UID: \"faa6a40d-2e9e-4348-a599-0df6f2d0177d\") " pod="openshift-console/console-f66c46c7c-skmxs"
Oct 30 04:44:12.791580 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.791568    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/faa6a40d-2e9e-4348-a599-0df6f2d0177d-service-ca\") pod \"console-f66c46c7c-skmxs\" (UID: \"faa6a40d-2e9e-4348-a599-0df6f2d0177d\") " pod="openshift-console/console-f66c46c7c-skmxs"
Oct 30 04:44:12.791751 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.791633    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/faa6a40d-2e9e-4348-a599-0df6f2d0177d-oauth-serving-cert\") pod \"console-f66c46c7c-skmxs\" (UID: \"faa6a40d-2e9e-4348-a599-0df6f2d0177d\") " pod="openshift-console/console-f66c46c7c-skmxs"
Oct 30 04:44:12.892748 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.892677    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/faa6a40d-2e9e-4348-a599-0df6f2d0177d-tmp\") pod \"console-f66c46c7c-skmxs\" (UID: \"faa6a40d-2e9e-4348-a599-0df6f2d0177d\") " pod="openshift-console/console-f66c46c7c-skmxs"
Oct 30 04:44:12.892748 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.892715    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/faa6a40d-2e9e-4348-a599-0df6f2d0177d-service-ca\") pod \"console-f66c46c7c-skmxs\" (UID: \"faa6a40d-2e9e-4348-a599-0df6f2d0177d\") " pod="openshift-console/console-f66c46c7c-skmxs"
Oct 30 04:44:12.892937 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.892760    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/faa6a40d-2e9e-4348-a599-0df6f2d0177d-oauth-serving-cert\") pod \"console-f66c46c7c-skmxs\" (UID: \"faa6a40d-2e9e-4348-a599-0df6f2d0177d\") " pod="openshift-console/console-f66c46c7c-skmxs"
Oct 30 04:44:12.892937 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.892779    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5q8q\" (UniqueName: \"kubernetes.io/projected/faa6a40d-2e9e-4348-a599-0df6f2d0177d-kube-api-access-p5q8q\") pod \"console-f66c46c7c-skmxs\" (UID: \"faa6a40d-2e9e-4348-a599-0df6f2d0177d\") " pod="openshift-console/console-f66c46c7c-skmxs"
Oct 30 04:44:12.892937 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.892808    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/faa6a40d-2e9e-4348-a599-0df6f2d0177d-console-serving-cert\") pod \"console-f66c46c7c-skmxs\" (UID: \"faa6a40d-2e9e-4348-a599-0df6f2d0177d\") " pod="openshift-console/console-f66c46c7c-skmxs"
Oct 30 04:44:12.893080 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.892945    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/faa6a40d-2e9e-4348-a599-0df6f2d0177d-console-config\") pod \"console-f66c46c7c-skmxs\" (UID: \"faa6a40d-2e9e-4348-a599-0df6f2d0177d\") " pod="openshift-console/console-f66c46c7c-skmxs"
Oct 30 04:44:12.893080 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.892996    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/faa6a40d-2e9e-4348-a599-0df6f2d0177d-console-oauth-config\") pod \"console-f66c46c7c-skmxs\" (UID: \"faa6a40d-2e9e-4348-a599-0df6f2d0177d\") " pod="openshift-console/console-f66c46c7c-skmxs"
Oct 30 04:44:12.893173 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.893094    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/faa6a40d-2e9e-4348-a599-0df6f2d0177d-tmp\") pod \"console-f66c46c7c-skmxs\" (UID: \"faa6a40d-2e9e-4348-a599-0df6f2d0177d\") " pod="openshift-console/console-f66c46c7c-skmxs"
Oct 30 04:44:12.893521 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.893504    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/faa6a40d-2e9e-4348-a599-0df6f2d0177d-oauth-serving-cert\") pod \"console-f66c46c7c-skmxs\" (UID: \"faa6a40d-2e9e-4348-a599-0df6f2d0177d\") " pod="openshift-console/console-f66c46c7c-skmxs"
Oct 30 04:44:12.893602 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.893531    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/faa6a40d-2e9e-4348-a599-0df6f2d0177d-service-ca\") pod \"console-f66c46c7c-skmxs\" (UID: \"faa6a40d-2e9e-4348-a599-0df6f2d0177d\") " pod="openshift-console/console-f66c46c7c-skmxs"
Oct 30 04:44:12.893602 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.893569    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/faa6a40d-2e9e-4348-a599-0df6f2d0177d-console-config\") pod \"console-f66c46c7c-skmxs\" (UID: \"faa6a40d-2e9e-4348-a599-0df6f2d0177d\") " pod="openshift-console/console-f66c46c7c-skmxs"
Oct 30 04:44:12.895123 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.895106    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/faa6a40d-2e9e-4348-a599-0df6f2d0177d-console-serving-cert\") pod \"console-f66c46c7c-skmxs\" (UID: \"faa6a40d-2e9e-4348-a599-0df6f2d0177d\") " pod="openshift-console/console-f66c46c7c-skmxs"
Oct 30 04:44:12.895208 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.895161    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/faa6a40d-2e9e-4348-a599-0df6f2d0177d-console-oauth-config\") pod \"console-f66c46c7c-skmxs\" (UID: \"faa6a40d-2e9e-4348-a599-0df6f2d0177d\") " pod="openshift-console/console-f66c46c7c-skmxs"
Oct 30 04:44:12.912413 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.912391    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5q8q\" (UniqueName: \"kubernetes.io/projected/faa6a40d-2e9e-4348-a599-0df6f2d0177d-kube-api-access-p5q8q\") pod \"console-f66c46c7c-skmxs\" (UID: \"faa6a40d-2e9e-4348-a599-0df6f2d0177d\") " pod="openshift-console/console-f66c46c7c-skmxs"
Oct 30 04:44:12.931472 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.931442    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9mr8m" event={"ID":"76081e78-ba20-4f7c-b825-a9e0da23c511","Type":"ContainerStarted","Data":"eca0ce845d650d056cc3459188acb61c3d2b0a202f448d4e3e7d6dac752daffb"}
Oct 30 04:44:12.931859 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.931508    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9mr8m" event={"ID":"76081e78-ba20-4f7c-b825-a9e0da23c511","Type":"ContainerStarted","Data":"6786027f6388fa7e5c9d0b35b39aab41769e8fab180070a5bdc4ba6e1634adb1"}
Oct 30 04:44:12.932530 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.932512    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-95b457c5b-n92bg" event={"ID":"fd380d55-276e-45eb-8191-c5940ed8385b","Type":"ContainerStarted","Data":"1f57f7ee285d7f2006cb35fd89c62f6d4029fb5a9c788ddc354a196be0931cd4"}
Oct 30 04:44:12.958249 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.958228    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f66c46c7c-skmxs"
Oct 30 04:44:12.965572 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.965551    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:12.977124 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:12.974013    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9mr8m" podStartSLOduration=3.064917358 podStartE2EDuration="3.973997821s" totalImagesPullingTime="842.323688ms" totalInitContainerRuntime="66.756775ms" isStatefulPod=true podCreationTimestamp="2026-10-30 04:44:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:44:12.972550197 +0000 UTC m=+53.904515924" watchObservedRunningTime="2026-10-30 04:44:12.973997821 +0000 UTC m=+53.905963549"
Oct 30 04:44:13.067388 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:13.067360    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:13.167499 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:13.167423    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:13.249530 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:13.249507    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f66c46c7c-skmxs"]
Oct 30 04:44:13.251792 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:44:13.251756    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaa6a40d_2e9e_4348_a599_0df6f2d0177d.slice/crio-1c2c7e85e42f36c1dd3132a5fc87e0b98b6d22ea9dec0a9c5cd39844c3050fa9 WatchSource:0}: Error finding container 1c2c7e85e42f36c1dd3132a5fc87e0b98b6d22ea9dec0a9c5cd39844c3050fa9: Status 404 returned error can't find the container with id 1c2c7e85e42f36c1dd3132a5fc87e0b98b6d22ea9dec0a9c5cd39844c3050fa9
Oct 30 04:44:13.253536 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:13.253510    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:13.938818 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:13.938784    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f66c46c7c-skmxs" event={"ID":"faa6a40d-2e9e-4348-a599-0df6f2d0177d","Type":"ContainerStarted","Data":"1c2c7e85e42f36c1dd3132a5fc87e0b98b6d22ea9dec0a9c5cd39844c3050fa9"}
Oct 30 04:44:14.462718 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:14.462655    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:14.515607 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:14.515573    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-r5tdl"]
Oct 30 04:44:14.519135 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:14.519109    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-r5tdl"
Oct 30 04:44:14.535498 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:14.535345    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-r5tdl"]
Oct 30 04:44:14.547549 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:14.547522    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-bxf29\""
Oct 30 04:44:14.547678 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:14.547578    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 30 04:44:14.553394 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:14.553369    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:14.668718 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:14.668683    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:14.708712 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:14.708673    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eddb252a-1213-4eb9-9219-5a0abedf5587-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-r5tdl\" (UID: \"eddb252a-1213-4eb9-9219-5a0abedf5587\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-r5tdl"
Oct 30 04:44:14.708878 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:14.708720    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klznl\" (UniqueName: \"kubernetes.io/projected/eddb252a-1213-4eb9-9219-5a0abedf5587-kube-api-access-klznl\") pod \"monitoring-plugin-77f4f76594-r5tdl\" (UID: \"eddb252a-1213-4eb9-9219-5a0abedf5587\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-r5tdl"
Oct 30 04:44:14.810064 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:14.809986    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eddb252a-1213-4eb9-9219-5a0abedf5587-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-r5tdl\" (UID: \"eddb252a-1213-4eb9-9219-5a0abedf5587\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-r5tdl"
Oct 30 04:44:14.810064 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:14.810022    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klznl\" (UniqueName: \"kubernetes.io/projected/eddb252a-1213-4eb9-9219-5a0abedf5587-kube-api-access-klznl\") pod \"monitoring-plugin-77f4f76594-r5tdl\" (UID: \"eddb252a-1213-4eb9-9219-5a0abedf5587\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-r5tdl"
Oct 30 04:44:14.815693 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:14.815664    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eddb252a-1213-4eb9-9219-5a0abedf5587-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-r5tdl\" (UID: \"eddb252a-1213-4eb9-9219-5a0abedf5587\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-r5tdl"
Oct 30 04:44:14.830394 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:14.830359    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klznl\" (UniqueName: \"kubernetes.io/projected/eddb252a-1213-4eb9-9219-5a0abedf5587-kube-api-access-klznl\") pod \"monitoring-plugin-77f4f76594-r5tdl\" (UID: \"eddb252a-1213-4eb9-9219-5a0abedf5587\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-r5tdl"
Oct 30 04:44:14.832828 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:14.832804    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-r5tdl"
Oct 30 04:44:14.860649 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:14.860619    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:44:15.136823 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:15.136700    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-r5tdl"]
Oct 30 04:44:15.143945 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:44:15.143918    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeddb252a_1213_4eb9_9219_5a0abedf5587.slice/crio-fdbac03ac556c63dcaef04cfa683bab4791e457aaa8db342b5538ceea9d38aba WatchSource:0}: Error finding container fdbac03ac556c63dcaef04cfa683bab4791e457aaa8db342b5538ceea9d38aba: Status 404 returned error can't find the container with id fdbac03ac556c63dcaef04cfa683bab4791e457aaa8db342b5538ceea9d38aba
Oct 30 04:44:15.950474 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:15.950223    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-95b457c5b-n92bg" event={"ID":"fd380d55-276e-45eb-8191-c5940ed8385b","Type":"ContainerStarted","Data":"42a64c2c8ec74bc0e577876dad1f5e21028abb29a5be4137233d38d72cc895e3"}
Oct 30 04:44:15.950474 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:15.950311    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-95b457c5b-n92bg" event={"ID":"fd380d55-276e-45eb-8191-c5940ed8385b","Type":"ContainerStarted","Data":"6e72cd5a9982c0fc740fff7555c18f1ce8d16857e849e502dbea5ea5b2acebe5"}
Oct 30 04:44:15.950474 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:15.950354    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-95b457c5b-n92bg" event={"ID":"fd380d55-276e-45eb-8191-c5940ed8385b","Type":"ContainerStarted","Data":"2b84efbf80c0e625571bb394882cebc52465af4f05be95e53c5d48d3f1fc2831"}
Oct 30 04:44:15.952745 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:15.952691    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-r5tdl" event={"ID":"eddb252a-1213-4eb9-9219-5a0abedf5587","Type":"ContainerStarted","Data":"fdbac03ac556c63dcaef04cfa683bab4791e457aaa8db342b5538ceea9d38aba"}
Oct 30 04:44:16.687324 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:16.687297    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:16.913776 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:16.913711    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-bbk54"
Oct 30 04:44:16.959442 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:16.959403    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-95b457c5b-n92bg" event={"ID":"fd380d55-276e-45eb-8191-c5940ed8385b","Type":"ContainerStarted","Data":"384de6e97daaf56a3999f9fbdae92f75feef4ee04e59183dd01ffbb5152f3f33"}
Oct 30 04:44:16.959620 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:16.959484    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-95b457c5b-n92bg" event={"ID":"fd380d55-276e-45eb-8191-c5940ed8385b","Type":"ContainerStarted","Data":"ddf6c51e183c0efe0b3fd8161b3b890c08306993c19f36e1b217d178453c378d"}
Oct 30 04:44:16.959620 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:16.959537    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-95b457c5b-n92bg" event={"ID":"fd380d55-276e-45eb-8191-c5940ed8385b","Type":"ContainerStarted","Data":"75648279ee9d11edbd179552157701e06756a888aa56a62779abe09a1eafe2b4"}
Oct 30 04:44:16.961391 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:16.961365    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-r5tdl" event={"ID":"eddb252a-1213-4eb9-9219-5a0abedf5587","Type":"ContainerStarted","Data":"36abc6a5bc72a10c93d7cb2d180905c5e29c8600a33406534571ec2895d803c8"}
Oct 30 04:44:16.963150 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:16.963125    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f66c46c7c-skmxs" event={"ID":"faa6a40d-2e9e-4348-a599-0df6f2d0177d","Type":"ContainerStarted","Data":"51265ebd62788e2d4bb19965906472dc9a0d2e30f50ba8139ca61f25c48493e7"}
Oct 30 04:44:16.964029 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:16.964013    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-95b457c5b-n92bg"
Oct 30 04:44:16.964318 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:16.964278    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-r5tdl"
Oct 30 04:44:16.970265 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:16.970246    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-r5tdl"
Oct 30 04:44:17.008131 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:17.008088    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-95b457c5b-n92bg" podStartSLOduration=2.7716103 podStartE2EDuration="6.008075693s" totalImagesPullingTime="3.236465393s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:44:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:44:17.006851895 +0000 UTC m=+57.938817621" watchObservedRunningTime="2026-10-30 04:44:17.008075693 +0000 UTC m=+57.940041420"
Oct 30 04:44:17.048791 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:17.048746    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-f66c46c7c-skmxs" podStartSLOduration=1.926983286 podStartE2EDuration="5.0487347s" totalImagesPullingTime="3.121751414s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:44:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:44:17.047836721 +0000 UTC m=+57.979802449" watchObservedRunningTime="2026-10-30 04:44:17.0487347 +0000 UTC m=+57.980700426"
Oct 30 04:44:17.087519 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:17.087356    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-r5tdl" podStartSLOduration=1.637036578 podStartE2EDuration="3.087338191s" totalImagesPullingTime="1.450301613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:44:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:44:17.086470295 +0000 UTC m=+58.018436023" watchObservedRunningTime="2026-10-30 04:44:17.087338191 +0000 UTC m=+58.019303922"
Oct 30 04:44:18.791045 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:18.791002    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:18.892026 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:18.891996    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:21.595690 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:21.595653    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:21.693672 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:21.693641    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:22.959080 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:22.959049    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-f66c46c7c-skmxs"
Oct 30 04:44:22.959517 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:22.959111    2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-f66c46c7c-skmxs"
Oct 30 04:44:22.964175 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:22.964148    2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-f66c46c7c-skmxs"
Oct 30 04:44:22.972536 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:22.972516    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-95b457c5b-n92bg"
Oct 30 04:44:22.985110 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:22.985085    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-f66c46c7c-skmxs"
Oct 30 04:44:23.433544 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:23.433516    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7db77977b9-2f8qv"]
Oct 30 04:44:23.437705 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:23.437683    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7db77977b9-2f8qv"
Oct 30 04:44:23.458144 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:23.458110    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7db77977b9-2f8qv"]
Oct 30 04:44:23.484167 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:23.484080    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 30 04:44:23.486723 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:23.484854    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-trusted-ca-bundle\") pod \"console-7db77977b9-2f8qv\" (UID: \"47f03fac-07ba-4d50-9336-c5af8b4fbcc3\") " pod="openshift-console/console-7db77977b9-2f8qv"
Oct 30 04:44:23.486723 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:23.484931    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-tmp\") pod \"console-7db77977b9-2f8qv\" (UID: \"47f03fac-07ba-4d50-9336-c5af8b4fbcc3\") " pod="openshift-console/console-7db77977b9-2f8qv"
Oct 30 04:44:23.486723 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:23.484966    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-oauth-serving-cert\") pod \"console-7db77977b9-2f8qv\" (UID: \"47f03fac-07ba-4d50-9336-c5af8b4fbcc3\") " pod="openshift-console/console-7db77977b9-2f8qv"
Oct 30 04:44:23.486723 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:23.484996    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-service-ca\") pod \"console-7db77977b9-2f8qv\" (UID: \"47f03fac-07ba-4d50-9336-c5af8b4fbcc3\") " pod="openshift-console/console-7db77977b9-2f8qv"
Oct 30 04:44:23.486723 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:23.485020    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-console-oauth-config\") pod \"console-7db77977b9-2f8qv\" (UID: \"47f03fac-07ba-4d50-9336-c5af8b4fbcc3\") " pod="openshift-console/console-7db77977b9-2f8qv"
Oct 30 04:44:23.486723 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:23.485050    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6rb2\" (UniqueName: \"kubernetes.io/projected/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-kube-api-access-z6rb2\") pod \"console-7db77977b9-2f8qv\" (UID: \"47f03fac-07ba-4d50-9336-c5af8b4fbcc3\") " pod="openshift-console/console-7db77977b9-2f8qv"
Oct 30 04:44:23.486723 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:23.485092    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-console-config\") pod \"console-7db77977b9-2f8qv\" (UID: \"47f03fac-07ba-4d50-9336-c5af8b4fbcc3\") " pod="openshift-console/console-7db77977b9-2f8qv"
Oct 30 04:44:23.486723 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:23.485146    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-console-serving-cert\") pod \"console-7db77977b9-2f8qv\" (UID: \"47f03fac-07ba-4d50-9336-c5af8b4fbcc3\") " pod="openshift-console/console-7db77977b9-2f8qv"
Oct 30 04:44:23.588239 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:23.588206    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-trusted-ca-bundle\") pod \"console-7db77977b9-2f8qv\" (UID: \"47f03fac-07ba-4d50-9336-c5af8b4fbcc3\") " pod="openshift-console/console-7db77977b9-2f8qv"
Oct 30 04:44:23.588424 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:23.588257    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-tmp\") pod \"console-7db77977b9-2f8qv\" (UID: \"47f03fac-07ba-4d50-9336-c5af8b4fbcc3\") " pod="openshift-console/console-7db77977b9-2f8qv"
Oct 30 04:44:23.588424 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:23.588274    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-oauth-serving-cert\") pod \"console-7db77977b9-2f8qv\" (UID: \"47f03fac-07ba-4d50-9336-c5af8b4fbcc3\") " pod="openshift-console/console-7db77977b9-2f8qv"
Oct 30 04:44:23.588424 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:23.588290    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-service-ca\") pod \"console-7db77977b9-2f8qv\" (UID: \"47f03fac-07ba-4d50-9336-c5af8b4fbcc3\") " pod="openshift-console/console-7db77977b9-2f8qv"
Oct 30 04:44:23.588424 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:23.588307    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-console-oauth-config\") pod \"console-7db77977b9-2f8qv\" (UID: \"47f03fac-07ba-4d50-9336-c5af8b4fbcc3\") " pod="openshift-console/console-7db77977b9-2f8qv"
Oct 30 04:44:23.588424 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:23.588355    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6rb2\" (UniqueName: \"kubernetes.io/projected/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-kube-api-access-z6rb2\") pod \"console-7db77977b9-2f8qv\" (UID: \"47f03fac-07ba-4d50-9336-c5af8b4fbcc3\") " pod="openshift-console/console-7db77977b9-2f8qv"
Oct 30 04:44:23.588424 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:23.588381    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-console-config\") pod \"console-7db77977b9-2f8qv\" (UID: \"47f03fac-07ba-4d50-9336-c5af8b4fbcc3\") " pod="openshift-console/console-7db77977b9-2f8qv"
Oct 30 04:44:23.588424 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:23.588419    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-console-serving-cert\") pod \"console-7db77977b9-2f8qv\" (UID: \"47f03fac-07ba-4d50-9336-c5af8b4fbcc3\") " pod="openshift-console/console-7db77977b9-2f8qv"
Oct 30 04:44:23.588763 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:23.588728    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-tmp\") pod \"console-7db77977b9-2f8qv\" (UID: \"47f03fac-07ba-4d50-9336-c5af8b4fbcc3\") " pod="openshift-console/console-7db77977b9-2f8qv"
Oct 30 04:44:23.589210 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:23.589181    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-oauth-serving-cert\") pod \"console-7db77977b9-2f8qv\" (UID: \"47f03fac-07ba-4d50-9336-c5af8b4fbcc3\") " pod="openshift-console/console-7db77977b9-2f8qv"
Oct 30 04:44:23.589323 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:23.589245    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-console-config\") pod \"console-7db77977b9-2f8qv\" (UID: \"47f03fac-07ba-4d50-9336-c5af8b4fbcc3\") " pod="openshift-console/console-7db77977b9-2f8qv"
Oct 30 04:44:23.589323 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:23.589267    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-trusted-ca-bundle\") pod \"console-7db77977b9-2f8qv\" (UID: \"47f03fac-07ba-4d50-9336-c5af8b4fbcc3\") " pod="openshift-console/console-7db77977b9-2f8qv"
Oct 30 04:44:23.589323 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:23.589245    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-service-ca\") pod \"console-7db77977b9-2f8qv\" (UID: \"47f03fac-07ba-4d50-9336-c5af8b4fbcc3\") " pod="openshift-console/console-7db77977b9-2f8qv"
Oct 30 04:44:23.590804 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:23.590777    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-console-oauth-config\") pod \"console-7db77977b9-2f8qv\" (UID: \"47f03fac-07ba-4d50-9336-c5af8b4fbcc3\") " pod="openshift-console/console-7db77977b9-2f8qv"
Oct 30 04:44:23.590885 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:23.590877    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-console-serving-cert\") pod \"console-7db77977b9-2f8qv\" (UID: \"47f03fac-07ba-4d50-9336-c5af8b4fbcc3\") " pod="openshift-console/console-7db77977b9-2f8qv"
Oct 30 04:44:23.595136 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:23.595111    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:23.608126 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:23.608104    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6rb2\" (UniqueName: \"kubernetes.io/projected/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-kube-api-access-z6rb2\") pod \"console-7db77977b9-2f8qv\" (UID: \"47f03fac-07ba-4d50-9336-c5af8b4fbcc3\") " pod="openshift-console/console-7db77977b9-2f8qv"
Oct 30 04:44:23.677847 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:23.677814    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:23.758962 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:23.758849    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7db77977b9-2f8qv"
Oct 30 04:44:24.060786 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:24.060717    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7db77977b9-2f8qv"]
Oct 30 04:44:24.063847 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:44:24.063801    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47f03fac_07ba_4d50_9336_c5af8b4fbcc3.slice/crio-55e9f36fd3d87ecd971b3336a06458ceae8c710341869364e8912f232c844aac WatchSource:0}: Error finding container 55e9f36fd3d87ecd971b3336a06458ceae8c710341869364e8912f232c844aac: Status 404 returned error can't find the container with id 55e9f36fd3d87ecd971b3336a06458ceae8c710341869364e8912f232c844aac
Oct 30 04:44:24.543048 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:24.543013    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:24.619538 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:24.619506    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:24.711700 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:24.711669    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:24.810535 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:24.810464    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:24.989218 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:24.989187    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7db77977b9-2f8qv" event={"ID":"47f03fac-07ba-4d50-9336-c5af8b4fbcc3","Type":"ContainerStarted","Data":"b70a5072438ea0d797b5909d950bb8908f13d596a25fa233a362e066026c2bb2"}
Oct 30 04:44:24.989423 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:24.989237    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7db77977b9-2f8qv" event={"ID":"47f03fac-07ba-4d50-9336-c5af8b4fbcc3","Type":"ContainerStarted","Data":"55e9f36fd3d87ecd971b3336a06458ceae8c710341869364e8912f232c844aac"}
Oct 30 04:44:25.029325 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:25.029280    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7db77977b9-2f8qv" podStartSLOduration=2.029267565 podStartE2EDuration="2.029267565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:44:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:44:25.029090557 +0000 UTC m=+65.961056284" watchObservedRunningTime="2026-10-30 04:44:25.029267565 +0000 UTC m=+65.961233292"
Oct 30 04:44:25.506133 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:25.506099    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/41a4ec4b-b407-4316-9238-dee6be432d3c-original-pull-secret\") pod \"global-pull-secret-syncer-tt5mj\" (UID: \"41a4ec4b-b407-4316-9238-dee6be432d3c\") " pod="kube-system/global-pull-secret-syncer-tt5mj"
Oct 30 04:44:25.506518 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:25.506156    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8x7k\" (UniqueName: \"kubernetes.io/projected/3b1cae8a-12cf-4b59-85f0-a63f324fc8e1-kube-api-access-f8x7k\") pod \"network-check-target-dgsdt\" (UID: \"3b1cae8a-12cf-4b59-85f0-a63f324fc8e1\") " pod="openshift-network-diagnostics/network-check-target-dgsdt"
Oct 30 04:44:25.506518 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:25.506177    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f75d879-5e31-41fd-838c-b1f0408cb45a-metrics-certs\") pod \"network-metrics-daemon-vjfrn\" (UID: \"7f75d879-5e31-41fd-838c-b1f0408cb45a\") " pod="openshift-multus/network-metrics-daemon-vjfrn"
Oct 30 04:44:25.533970 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:25.533945    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 04:44:25.534066 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:25.533979    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 04:44:25.534066 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:25.533983    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 04:44:25.539221 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:25.539193    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f75d879-5e31-41fd-838c-b1f0408cb45a-metrics-certs\") pod \"network-metrics-daemon-vjfrn\" (UID: \"7f75d879-5e31-41fd-838c-b1f0408cb45a\") " pod="openshift-multus/network-metrics-daemon-vjfrn"
Oct 30 04:44:25.539221 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:25.539210    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/41a4ec4b-b407-4316-9238-dee6be432d3c-original-pull-secret\") pod \"global-pull-secret-syncer-tt5mj\" (UID: \"41a4ec4b-b407-4316-9238-dee6be432d3c\") " pod="kube-system/global-pull-secret-syncer-tt5mj"
Oct 30 04:44:25.551558 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:25.551540    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 04:44:25.559929 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:25.559891    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8x7k\" (UniqueName: \"kubernetes.io/projected/3b1cae8a-12cf-4b59-85f0-a63f324fc8e1-kube-api-access-f8x7k\") pod \"network-check-target-dgsdt\" (UID: \"3b1cae8a-12cf-4b59-85f0-a63f324fc8e1\") " pod="openshift-network-diagnostics/network-check-target-dgsdt"
Oct 30 04:44:25.580987 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:25.580966    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mxnkb\""
Oct 30 04:44:25.587372 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:25.587358    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tt5mj"
Oct 30 04:44:25.600475 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:25.597786    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-56dd5\""
Oct 30 04:44:25.600475 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:25.598666    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k257x\""
Oct 30 04:44:25.604874 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:25.604855    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vjfrn"
Oct 30 04:44:25.606044 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:25.606026    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgsdt"
Oct 30 04:44:25.881875 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:25.881848    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tt5mj"]
Oct 30 04:44:25.883137 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:44:25.883114    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41a4ec4b_b407_4316_9238_dee6be432d3c.slice/crio-06617a34cae62d587573f703a0f9751f896e6ffbd092ce68c4cb339ca96b4c6a WatchSource:0}: Error finding container 06617a34cae62d587573f703a0f9751f896e6ffbd092ce68c4cb339ca96b4c6a: Status 404 returned error can't find the container with id 06617a34cae62d587573f703a0f9751f896e6ffbd092ce68c4cb339ca96b4c6a
Oct 30 04:44:25.917264 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:25.917240    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vjfrn"]
Oct 30 04:44:25.917759 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:44:25.917738    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b1cae8a_12cf_4b59_85f0_a63f324fc8e1.slice/crio-e25a610fa35edc5ec7910b2cd0e9454bdbd9dd7942289d7bfea79b931518573b WatchSource:0}: Error finding container e25a610fa35edc5ec7910b2cd0e9454bdbd9dd7942289d7bfea79b931518573b: Status 404 returned error can't find the container with id e25a610fa35edc5ec7910b2cd0e9454bdbd9dd7942289d7bfea79b931518573b
Oct 30 04:44:25.918450 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:25.918431    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dgsdt"]
Oct 30 04:44:25.918864 ip-10-0-135-98 kubenswrapper[2579]: W1030 04:44:25.918837    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f75d879_5e31_41fd_838c_b1f0408cb45a.slice/crio-ae34671c26d15cc2d08cba5274e6c4ba4ebae35967ce18ff62ae9a627b41cfd5 WatchSource:0}: Error finding container ae34671c26d15cc2d08cba5274e6c4ba4ebae35967ce18ff62ae9a627b41cfd5: Status 404 returned error can't find the container with id ae34671c26d15cc2d08cba5274e6c4ba4ebae35967ce18ff62ae9a627b41cfd5
Oct 30 04:44:25.993786 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:25.993760    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dgsdt" event={"ID":"3b1cae8a-12cf-4b59-85f0-a63f324fc8e1","Type":"ContainerStarted","Data":"e25a610fa35edc5ec7910b2cd0e9454bdbd9dd7942289d7bfea79b931518573b"}
Oct 30 04:44:25.994796 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:25.994771    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vjfrn" event={"ID":"7f75d879-5e31-41fd-838c-b1f0408cb45a","Type":"ContainerStarted","Data":"ae34671c26d15cc2d08cba5274e6c4ba4ebae35967ce18ff62ae9a627b41cfd5"}
Oct 30 04:44:25.995796 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:25.995774    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tt5mj" event={"ID":"41a4ec4b-b407-4316-9238-dee6be432d3c","Type":"ContainerStarted","Data":"06617a34cae62d587573f703a0f9751f896e6ffbd092ce68c4cb339ca96b4c6a"}
Oct 30 04:44:26.088879 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:26.088733    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:26.192775 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:26.192694    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:27.240307 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:27.240132    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:27.344118 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:27.344049    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:28.008791 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:28.008754    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vjfrn" event={"ID":"7f75d879-5e31-41fd-838c-b1f0408cb45a","Type":"ContainerStarted","Data":"fdb23a2a847ad5ce39aa2a0f61db73dc94ea782b9071053268543b8f84695819"}
Oct 30 04:44:28.008994 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:28.008807    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vjfrn" event={"ID":"7f75d879-5e31-41fd-838c-b1f0408cb45a","Type":"ContainerStarted","Data":"e535ed0540b113c278f0ec5036bde7d4aa57120286ff19f622d171ed19f95134"}
Oct 30 04:44:28.052973 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:28.052892    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-vjfrn" podStartSLOduration=67.994473938 podStartE2EDuration="1m9.05287342s" totalImagesPullingTime="1.058399482s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:43:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:44:28.052280375 +0000 UTC m=+68.984246103" watchObservedRunningTime="2026-10-30 04:44:28.05287342 +0000 UTC m=+68.984839150"
Oct 30 04:44:30.019296 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:30.019256    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dgsdt" event={"ID":"3b1cae8a-12cf-4b59-85f0-a63f324fc8e1","Type":"ContainerStarted","Data":"7d5b04908d7fbc7fa10a3c910bcdab31bd7859ba8b3af1a0e1ff1cc340d4a99c"}
Oct 30 04:44:30.019763 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:30.019411    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dgsdt"
Oct 30 04:44:30.059280 ip-10-0-135-98 kubenswrapper[2579]: I1030 04:44:30.059226    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-dgsdt" podStartSLOduration=67.965512659 podStartE2EDuration="1m11.059207257s" totalImagesPullingTime="3.093694598s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:43:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:44:30.058997317 +0000 UTC m=+70.990963046" watchObservedRunningTime="2026-10-30 04:44:30.059207257 +0000 UTC m=+70.991172986"
Oct 30 04:44:30.716540 ip-10-0-135-98 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 04:44:30.733283 ip-10-0-135-98 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 04:44:30.733461 ip-10-0-135-98 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 04:44:30.733512 ip-10-0-135-98 systemd[1]: kubelet.service: Consumed 3.825s CPU time.
Oct 30 04:44:30.745656 ip-10-0-135-98 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 04:44:30.805218 ip-10-0-135-98 kubenswrapper[5898]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 04:44:30.805218 ip-10-0-135-98 kubenswrapper[5898]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 04:44:30.805218 ip-10-0-135-98 kubenswrapper[5898]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 04:44:30.805218 ip-10-0-135-98 kubenswrapper[5898]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 04:44:30.807391 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807313    5898 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 04:44:30.807391 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807388    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 04:44:30.807391 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807392    5898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 04:44:30.807391 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807395    5898 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 04:44:30.807391 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807398    5898 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 04:44:30.807594 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807402    5898 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 04:44:30.807594 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807405    5898 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 04:44:30.807594 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807408    5898 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 04:44:30.807594 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807411    5898 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 04:44:30.807594 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807413    5898 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 04:44:30.807594 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807416    5898 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 04:44:30.807594 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807419    5898 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 04:44:30.807594 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807422    5898 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 04:44:30.807594 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807425    5898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 04:44:30.807594 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807434    5898 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 04:44:30.807594 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807437    5898 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 04:44:30.807594 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807440    5898 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 04:44:30.807594 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807442    5898 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 04:44:30.807594 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807445    5898 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 04:44:30.807594 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807448    5898 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 04:44:30.807594 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807451    5898 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 04:44:30.807594 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807453    5898 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 04:44:30.807594 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807456    5898 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 04:44:30.807594 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807459    5898 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 04:44:30.807594 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807461    5898 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 04:44:30.808135 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807464    5898 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 04:44:30.808135 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807467    5898 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 04:44:30.808135 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807470    5898 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 04:44:30.808135 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807473    5898 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 04:44:30.808135 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807476    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 04:44:30.808135 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807479    5898 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 04:44:30.808135 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807482    5898 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 04:44:30.808135 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807485    5898 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 04:44:30.808135 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807487    5898 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 04:44:30.808135 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807490    5898 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 04:44:30.808135 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807493    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 04:44:30.808135 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807495    5898 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 04:44:30.808135 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807498    5898 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 04:44:30.808135 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807500    5898 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 04:44:30.808135 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807503    5898 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 04:44:30.808135 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807505    5898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 04:44:30.808135 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807508    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 04:44:30.808135 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807518    5898 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 04:44:30.808135 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807522    5898 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 04:44:30.808607 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807527    5898 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 04:44:30.808607 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807534    5898 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 04:44:30.808607 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807539    5898 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 04:44:30.808607 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807544    5898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 04:44:30.808607 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807548    5898 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 04:44:30.808607 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807553    5898 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 04:44:30.808607 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807557    5898 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 04:44:30.808607 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807561    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 04:44:30.808607 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807565    5898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 04:44:30.808607 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807570    5898 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 04:44:30.808607 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807574    5898 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 04:44:30.808607 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807577    5898 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 04:44:30.808607 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807580    5898 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 04:44:30.808607 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807583    5898 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 04:44:30.808607 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807587    5898 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 04:44:30.808607 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807590    5898 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 04:44:30.808607 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807592    5898 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 04:44:30.808607 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807595    5898 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 04:44:30.808607 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807597    5898 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 04:44:30.809115 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807600    5898 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 04:44:30.809115 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807603    5898 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 04:44:30.809115 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807606    5898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 04:44:30.809115 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807608    5898 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 04:44:30.809115 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807611    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 04:44:30.809115 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807614    5898 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 04:44:30.809115 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807617    5898 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 04:44:30.809115 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807621    5898 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 04:44:30.809115 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807624    5898 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 04:44:30.809115 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807626    5898 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 04:44:30.809115 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807629    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 04:44:30.809115 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807631    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 04:44:30.809115 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807634    5898 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 04:44:30.809115 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807636    5898 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 04:44:30.809115 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807639    5898 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 04:44:30.809115 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807642    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 04:44:30.809115 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807645    5898 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 04:44:30.809115 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807648    5898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 04:44:30.809115 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807650    5898 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 04:44:30.809115 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807653    5898 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 04:44:30.809680 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807655    5898 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 04:44:30.809680 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807658    5898 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 04:44:30.809680 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807660    5898 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 04:44:30.809680 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807663    5898 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 04:44:30.809680 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807666    5898 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 04:44:30.809680 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807668    5898 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 04:44:30.809680 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807671    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 04:44:30.809680 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807674    5898 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 04:44:30.809680 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807677    5898 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 04:44:30.809680 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807679    5898 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 04:44:30.809680 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807682    5898 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 04:44:30.809680 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807684    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 04:44:30.809680 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807688    5898 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 04:44:30.809680 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807691    5898 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 04:44:30.809680 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807693    5898 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 04:44:30.809680 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807696    5898 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 04:44:30.809680 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807699    5898 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 04:44:30.809680 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807702    5898 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 04:44:30.809680 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807705    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 04:44:30.809680 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807707    5898 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 04:44:30.810263 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807712    5898 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 04:44:30.810263 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807716    5898 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 04:44:30.810263 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.807720    5898 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 04:44:30.810263 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808529    5898 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 04:44:30.810263 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808537    5898 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 04:44:30.810263 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808540    5898 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 04:44:30.810263 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808544    5898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 04:44:30.810263 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808547    5898 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 04:44:30.810263 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808555    5898 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 04:44:30.810263 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808558    5898 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 04:44:30.810263 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808561    5898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 04:44:30.810263 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808564    5898 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 04:44:30.810263 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808567    5898 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 04:44:30.810263 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808569    5898 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 04:44:30.810263 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808572    5898 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 04:44:30.810263 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808574    5898 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 04:44:30.810263 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808577    5898 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 04:44:30.810263 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808580    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 04:44:30.810725 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808582    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 04:44:30.810725 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808585    5898 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 04:44:30.810725 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808588    5898 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 04:44:30.810725 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808591    5898 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 04:44:30.810725 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808594    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 04:44:30.810725 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808596    5898 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 04:44:30.810725 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808599    5898 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 04:44:30.810725 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808601    5898 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 04:44:30.810725 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808604    5898 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 04:44:30.810725 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808606    5898 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 04:44:30.810725 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808609    5898 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 04:44:30.810725 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808612    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 04:44:30.810725 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808614    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 04:44:30.810725 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808617    5898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 04:44:30.810725 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808620    5898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 04:44:30.810725 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808623    5898 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 04:44:30.810725 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808626    5898 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 04:44:30.810725 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808629    5898 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 04:44:30.810725 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808631    5898 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 04:44:30.810725 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808633    5898 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 04:44:30.811256 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808636    5898 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 04:44:30.811256 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808639    5898 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 04:44:30.811256 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808641    5898 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 04:44:30.811256 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808646    5898 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 04:44:30.811256 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808649    5898 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 04:44:30.811256 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808652    5898 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 04:44:30.811256 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808655    5898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 04:44:30.811256 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808657    5898 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 04:44:30.811256 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808660    5898 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 04:44:30.811256 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808663    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 04:44:30.811256 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808666    5898 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 04:44:30.811256 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808669    5898 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 04:44:30.811256 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808671    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 04:44:30.811256 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808675    5898 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 04:44:30.811256 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808677    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 04:44:30.811256 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808680    5898 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 04:44:30.811256 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808683    5898 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 04:44:30.811256 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808685    5898 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 04:44:30.811256 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808688    5898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 04:44:30.811729 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808691    5898 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 04:44:30.811729 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808693    5898 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 04:44:30.811729 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808696    5898 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 04:44:30.811729 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808699    5898 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 04:44:30.811729 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808701    5898 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 04:44:30.811729 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808704    5898 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 04:44:30.811729 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808706    5898 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 04:44:30.811729 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808709    5898 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 04:44:30.811729 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808712    5898 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 04:44:30.811729 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808714    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 04:44:30.811729 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808717    5898 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 04:44:30.811729 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808720    5898 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 04:44:30.811729 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808722    5898 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 04:44:30.811729 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808725    5898 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 04:44:30.811729 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808732    5898 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 04:44:30.811729 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808735    5898 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 04:44:30.811729 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808737    5898 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 04:44:30.811729 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808740    5898 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 04:44:30.811729 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808743    5898 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 04:44:30.811729 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808746    5898 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 04:44:30.812238 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808748    5898 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 04:44:30.812238 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808751    5898 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 04:44:30.812238 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808753    5898 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 04:44:30.812238 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808757    5898 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 04:44:30.812238 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808761    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 04:44:30.812238 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808764    5898 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 04:44:30.812238 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808768    5898 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 04:44:30.812238 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808770    5898 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 04:44:30.812238 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808773    5898 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 04:44:30.812238 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808775    5898 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 04:44:30.812238 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808778    5898 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 04:44:30.812238 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808780    5898 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 04:44:30.812238 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808783    5898 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 04:44:30.812238 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808785    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 04:44:30.812238 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808788    5898 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 04:44:30.812238 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808791    5898 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 04:44:30.812238 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808794    5898 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 04:44:30.812238 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808796    5898 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 04:44:30.812238 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808799    5898 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 04:44:30.812238 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808801    5898 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 04:44:30.812772 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808804    5898 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 04:44:30.812772 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808806    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 04:44:30.812772 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808809    5898 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 04:44:30.812772 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808813    5898 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 04:44:30.812772 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808817    5898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 04:44:30.812772 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808819    5898 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 04:44:30.812772 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808824    5898 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 04:44:30.812772 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808827    5898 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 04:44:30.812772 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808829    5898 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 04:44:30.812772 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808832    5898 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 04:44:30.812772 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808835    5898 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 04:44:30.812772 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.808838    5898 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 04:44:30.812772 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809580    5898 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 04:44:30.812772 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809589    5898 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 04:44:30.812772 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809593    5898 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 04:44:30.812772 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809597    5898 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 04:44:30.812772 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809600    5898 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 04:44:30.812772 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809602    5898 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 04:44:30.812772 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809605    5898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 04:44:30.813324 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809608    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 04:44:30.813324 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809611    5898 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 04:44:30.813324 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809613    5898 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 04:44:30.813324 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809616    5898 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 04:44:30.813324 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809619    5898 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 04:44:30.813324 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809621    5898 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 04:44:30.813324 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809624    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 04:44:30.813324 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809627    5898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 04:44:30.813324 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809630    5898 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 04:44:30.813324 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809632    5898 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 04:44:30.813324 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809635    5898 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 04:44:30.813324 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809638    5898 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 04:44:30.813324 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809642    5898 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 04:44:30.813324 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809646    5898 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 04:44:30.813324 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809649    5898 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 04:44:30.813324 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809652    5898 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 04:44:30.813324 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809655    5898 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 04:44:30.813324 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809657    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 04:44:30.813324 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809660    5898 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 04:44:30.813324 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809665    5898 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 04:44:30.813849 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809668    5898 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 04:44:30.813849 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809670    5898 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 04:44:30.813849 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809673    5898 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 04:44:30.813849 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809676    5898 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 04:44:30.813849 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809679    5898 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 04:44:30.813849 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809682    5898 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 04:44:30.813849 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809686    5898 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 04:44:30.813849 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809690    5898 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 04:44:30.813849 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809693    5898 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 04:44:30.813849 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809696    5898 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 04:44:30.813849 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809699    5898 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 04:44:30.813849 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809702    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 04:44:30.813849 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809706    5898 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 04:44:30.813849 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809709    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 04:44:30.813849 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809711    5898 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 04:44:30.813849 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809714    5898 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 04:44:30.813849 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809717    5898 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 04:44:30.813849 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809719    5898 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 04:44:30.813849 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809722    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 04:44:30.814379 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809727    5898 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 04:44:30.814379 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809731    5898 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 04:44:30.814379 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809735    5898 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 04:44:30.814379 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809739    5898 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 04:44:30.814379 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809742    5898 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 04:44:30.814379 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809745    5898 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 04:44:30.814379 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809748    5898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 04:44:30.814379 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809750    5898 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 04:44:30.814379 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809753    5898 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 04:44:30.814379 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809756    5898 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 04:44:30.814379 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809759    5898 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 04:44:30.814379 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809761    5898 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 04:44:30.814379 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809764    5898 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 04:44:30.814379 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809767    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 04:44:30.814379 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809770    5898 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 04:44:30.814379 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809772    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 04:44:30.814379 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809775    5898 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 04:44:30.814379 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809778    5898 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 04:44:30.814379 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809781    5898 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 04:44:30.814866 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809784    5898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 04:44:30.814866 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809787    5898 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 04:44:30.814866 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809789    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 04:44:30.814866 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809792    5898 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 04:44:30.814866 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809796    5898 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 04:44:30.814866 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809801    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 04:44:30.814866 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809805    5898 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 04:44:30.814866 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809809    5898 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 04:44:30.814866 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809812    5898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 04:44:30.814866 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809814    5898 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 04:44:30.814866 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809817    5898 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 04:44:30.814866 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809819    5898 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 04:44:30.814866 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809822    5898 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 04:44:30.814866 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809824    5898 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 04:44:30.814866 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809827    5898 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 04:44:30.814866 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809830    5898 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 04:44:30.814866 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809832    5898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 04:44:30.814866 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809835    5898 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 04:44:30.814866 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809837    5898 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 04:44:30.814866 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809840    5898 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 04:44:30.815366 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809843    5898 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 04:44:30.815366 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809845    5898 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 04:44:30.815366 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809848    5898 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 04:44:30.815366 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809850    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 04:44:30.815366 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809853    5898 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 04:44:30.815366 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809857    5898 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 04:44:30.815366 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809860    5898 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 04:44:30.815366 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809863    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 04:44:30.815366 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809865    5898 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 04:44:30.815366 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809868    5898 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 04:44:30.815366 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809870    5898 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 04:44:30.815366 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809874    5898 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 04:44:30.815366 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809877    5898 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 04:44:30.815366 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809879    5898 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 04:44:30.815366 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809881    5898 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 04:44:30.815366 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809884    5898 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 04:44:30.815366 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809886    5898 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 04:44:30.815366 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809889    5898 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 04:44:30.815366 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809892    5898 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 04:44:30.815366 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809895    5898 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 04:44:30.815864 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.809913    5898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 04:44:30.815864 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810006    5898 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 04:44:30.815864 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810014    5898 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 04:44:30.815864 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810020    5898 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 04:44:30.815864 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810024    5898 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 04:44:30.815864 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810028    5898 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 04:44:30.815864 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810032    5898 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 04:44:30.815864 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810036    5898 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 04:44:30.815864 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810040    5898 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 04:44:30.815864 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810044    5898 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 04:44:30.815864 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810047    5898 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 04:44:30.815864 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810050    5898 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 04:44:30.815864 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810054    5898 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 04:44:30.815864 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810057    5898 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 04:44:30.815864 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810060    5898 flags.go:64] FLAG: --cgroup-root=""
Oct 30 04:44:30.815864 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810063    5898 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 04:44:30.815864 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810066    5898 flags.go:64] FLAG: --client-ca-file=""
Oct 30 04:44:30.815864 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810071    5898 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 04:44:30.815864 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810074    5898 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 04:44:30.815864 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810079    5898 flags.go:64] FLAG: --cluster-domain=""
Oct 30 04:44:30.815864 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810082    5898 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 04:44:30.815864 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810085    5898 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 04:44:30.815864 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810088    5898 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 04:44:30.815864 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810092    5898 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 04:44:30.816489 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810096    5898 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 04:44:30.816489 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810099    5898 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 04:44:30.816489 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810102    5898 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 04:44:30.816489 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810106    5898 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 04:44:30.816489 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810109    5898 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 04:44:30.816489 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810112    5898 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 04:44:30.816489 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810115    5898 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 04:44:30.816489 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810118    5898 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 04:44:30.816489 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810121    5898 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 04:44:30.816489 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810125    5898 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 04:44:30.816489 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810129    5898 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 04:44:30.816489 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810131    5898 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 04:44:30.816489 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810134    5898 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 04:44:30.816489 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810138    5898 flags.go:64] FLAG: --enable-server="true"
Oct 30 04:44:30.816489 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810141    5898 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 04:44:30.816489 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810145    5898 flags.go:64] FLAG: --event-burst="100"
Oct 30 04:44:30.816489 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810148    5898 flags.go:64] FLAG: --event-qps="50"
Oct 30 04:44:30.816489 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810151    5898 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 04:44:30.816489 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810157    5898 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 04:44:30.816489 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810160    5898 flags.go:64] FLAG: --eviction-hard=""
Oct 30 04:44:30.816489 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810165    5898 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 04:44:30.816489 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810168    5898 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 04:44:30.816489 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810171    5898 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 04:44:30.816489 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810174    5898 flags.go:64] FLAG: --eviction-soft=""
Oct 30 04:44:30.816489 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810177    5898 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 04:44:30.817319 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810181    5898 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 04:44:30.817319 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810184    5898 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 04:44:30.817319 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810187    5898 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 04:44:30.817319 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810190    5898 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 04:44:30.817319 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810193    5898 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 04:44:30.817319 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810196    5898 flags.go:64] FLAG: --feature-gates=""
Oct 30 04:44:30.817319 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810201    5898 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 04:44:30.817319 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810205    5898 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 04:44:30.817319 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810209    5898 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 04:44:30.817319 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810212    5898 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 04:44:30.817319 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810215    5898 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 04:44:30.817319 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810218    5898 flags.go:64] FLAG: --help="false"
Oct 30 04:44:30.817319 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810221    5898 flags.go:64] FLAG: --hostname-override="ip-10-0-135-98.ec2.internal"
Oct 30 04:44:30.817319 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810225    5898 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 04:44:30.817319 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810227    5898 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 04:44:30.817319 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810230    5898 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 04:44:30.817319 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810234    5898 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 04:44:30.817319 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810237    5898 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 04:44:30.817319 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810240    5898 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 04:44:30.817319 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810243    5898 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 04:44:30.817319 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810246    5898 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 04:44:30.817319 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810249    5898 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 04:44:30.817319 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810252    5898 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 04:44:30.818318 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810255    5898 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 04:44:30.818318 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810258    5898 flags.go:64] FLAG: --kube-reserved=""
Oct 30 04:44:30.818318 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810262    5898 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 04:44:30.818318 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810265    5898 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 04:44:30.818318 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810269    5898 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 04:44:30.818318 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810272    5898 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 04:44:30.818318 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810275    5898 flags.go:64] FLAG: --lock-file=""
Oct 30 04:44:30.818318 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810278    5898 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 04:44:30.818318 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810281    5898 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 04:44:30.818318 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810285    5898 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 04:44:30.818318 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810290    5898 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 04:44:30.818318 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810293    5898 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 04:44:30.818318 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810296    5898 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 04:44:30.818318 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810300    5898 flags.go:64] FLAG: --logging-format="text"
Oct 30 04:44:30.818318 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810303    5898 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 04:44:30.818318 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810306    5898 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 04:44:30.818318 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810309    5898 flags.go:64] FLAG: --manifest-url=""
Oct 30 04:44:30.818318 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810312    5898 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 04:44:30.818318 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810317    5898 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 04:44:30.818318 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810321    5898 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 04:44:30.818318 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810325    5898 flags.go:64] FLAG: --max-pods="110"
Oct 30 04:44:30.818318 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810328    5898 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 04:44:30.818318 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810331    5898 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 04:44:30.818318 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810334    5898 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 04:44:30.818318 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810337    5898 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 04:44:30.818954 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810340    5898 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 04:44:30.818954 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810343    5898 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 04:44:30.818954 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810346    5898 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 04:44:30.818954 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810353    5898 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 04:44:30.818954 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810356    5898 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 04:44:30.818954 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810359    5898 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 04:44:30.818954 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810363    5898 flags.go:64] FLAG: --pod-cidr=""
Oct 30 04:44:30.818954 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810365    5898 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 04:44:30.818954 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810368    5898 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 04:44:30.818954 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810373    5898 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 04:44:30.818954 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810376    5898 flags.go:64] FLAG: --port="10250"
Oct 30 04:44:30.818954 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810380    5898 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 04:44:30.818954 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810382    5898 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0207b1bc66b173d2c"
Oct 30 04:44:30.818954 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810386    5898 flags.go:64] FLAG: --qos-reserved=""
Oct 30 04:44:30.818954 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810389    5898 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 04:44:30.818954 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810392    5898 flags.go:64] FLAG: --register-node="true"
Oct 30 04:44:30.818954 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810396    5898 flags.go:64] FLAG: --register-with-taints=""
Oct 30 04:44:30.818954 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810400    5898 flags.go:64] FLAG: --registry-burst="10"
Oct 30 04:44:30.818954 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810403    5898 flags.go:64] FLAG: --registry-qps="5"
Oct 30 04:44:30.818954 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810406    5898 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 04:44:30.818954 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810409    5898 flags.go:64] FLAG: --reserved-memory=""
Oct 30 04:44:30.818954 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810413    5898 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 04:44:30.818954 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810416    5898 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 04:44:30.818954 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810419    5898 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 04:44:30.818954 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810422    5898 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 04:44:30.818954 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810425    5898 flags.go:64] FLAG: --runonce="false"
Oct 30 04:44:30.819606 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810429    5898 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 04:44:30.819606 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810432    5898 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 04:44:30.819606 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810435    5898 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 04:44:30.819606 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810438    5898 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 04:44:30.819606 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810441    5898 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 04:44:30.819606 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810444    5898 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 04:44:30.819606 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810447    5898 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 04:44:30.819606 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810450    5898 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 04:44:30.819606 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810453    5898 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 04:44:30.819606 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810456    5898 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 04:44:30.819606 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810460    5898 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 04:44:30.819606 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810462    5898 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 04:44:30.819606 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810466    5898 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 04:44:30.819606 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810468    5898 flags.go:64] FLAG: --system-cgroups=""
Oct 30 04:44:30.819606 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810471    5898 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 04:44:30.819606 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810478    5898 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 04:44:30.819606 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810481    5898 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 04:44:30.819606 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810484    5898 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 04:44:30.819606 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810489    5898 flags.go:64] FLAG: --tls-min-version=""
Oct 30 04:44:30.819606 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810492    5898 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 04:44:30.819606 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810495    5898 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 04:44:30.819606 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810498    5898 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 04:44:30.819606 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810502    5898 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 04:44:30.819606 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810505    5898 flags.go:64] FLAG: --v="2"
Oct 30 04:44:30.819606 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810510    5898 flags.go:64] FLAG: --version="false"
Oct 30 04:44:30.820328 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810514    5898 flags.go:64] FLAG: --vmodule=""
Oct 30 04:44:30.820328 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810518    5898 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 04:44:30.820328 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810521    5898 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 04:44:30.820328 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810656    5898 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 04:44:30.820328 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810660    5898 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 04:44:30.820328 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810664    5898 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 04:44:30.820328 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810666    5898 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 04:44:30.820328 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810669    5898 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 04:44:30.820328 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810673    5898 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 04:44:30.820328 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810675    5898 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 04:44:30.820328 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810678    5898 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 04:44:30.820328 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810681    5898 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 04:44:30.820328 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810684    5898 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 04:44:30.820328 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810686    5898 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 04:44:30.820328 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810689    5898 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 04:44:30.820328 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810692    5898 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 04:44:30.820328 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810694    5898 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 04:44:30.820328 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810697    5898 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 04:44:30.820328 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810700    5898 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 04:44:30.820328 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810702    5898 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 04:44:30.820862 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810705    5898 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 04:44:30.820862 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810707    5898 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 04:44:30.820862 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810712    5898 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 04:44:30.820862 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810714    5898 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 04:44:30.820862 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810717    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 04:44:30.820862 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810719    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 04:44:30.820862 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810722    5898 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 04:44:30.820862 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810725    5898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 04:44:30.820862 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810727    5898 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 04:44:30.820862 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810731    5898 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 04:44:30.820862 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810734    5898 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 04:44:30.820862 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810737    5898 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 04:44:30.820862 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810740    5898 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 04:44:30.820862 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810742    5898 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 04:44:30.820862 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810745    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 04:44:30.820862 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810749    5898 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 04:44:30.820862 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810753    5898 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 04:44:30.820862 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810756    5898 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 04:44:30.820862 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810759    5898 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 04:44:30.821383 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810762    5898 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 04:44:30.821383 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810765    5898 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 04:44:30.821383 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810768    5898 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 04:44:30.821383 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810770    5898 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 04:44:30.821383 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810773    5898 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 04:44:30.821383 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810775    5898 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 04:44:30.821383 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810778    5898 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 04:44:30.821383 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810780    5898 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 04:44:30.821383 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810783    5898 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 04:44:30.821383 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810785    5898 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 04:44:30.821383 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810788    5898 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 04:44:30.821383 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810791    5898 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 04:44:30.821383 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810793    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 04:44:30.821383 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810796    5898 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 04:44:30.821383 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810799    5898 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 04:44:30.821383 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810803    5898 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 04:44:30.821383 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810805    5898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 04:44:30.821383 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810808    5898 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 04:44:30.821383 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810810    5898 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 04:44:30.821383 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810813    5898 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 04:44:30.821993 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810815    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 04:44:30.821993 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810818    5898 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 04:44:30.821993 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810822    5898 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 04:44:30.821993 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810824    5898 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 04:44:30.821993 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810827    5898 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 04:44:30.821993 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810830    5898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 04:44:30.821993 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810832    5898 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 04:44:30.821993 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810835    5898 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 04:44:30.821993 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810837    5898 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 04:44:30.821993 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810840    5898 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 04:44:30.821993 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810844    5898 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 04:44:30.821993 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810848    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 04:44:30.821993 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810851    5898 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 04:44:30.821993 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810854    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 04:44:30.821993 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810857    5898 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 04:44:30.821993 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810860    5898 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 04:44:30.821993 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810863    5898 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 04:44:30.821993 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810865    5898 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 04:44:30.821993 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810868    5898 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 04:44:30.821993 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810870    5898 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 04:44:30.822619 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810873    5898 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 04:44:30.822619 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810876    5898 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 04:44:30.822619 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810878    5898 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 04:44:30.822619 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810881    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 04:44:30.822619 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810883    5898 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 04:44:30.822619 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810886    5898 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 04:44:30.822619 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810888    5898 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 04:44:30.822619 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810892    5898 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 04:44:30.822619 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810895    5898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 04:44:30.822619 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810913    5898 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 04:44:30.822619 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810916    5898 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 04:44:30.822619 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810919    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 04:44:30.822619 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810921    5898 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 04:44:30.822619 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810924    5898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 04:44:30.822619 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810928    5898 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 04:44:30.822619 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810931    5898 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 04:44:30.822619 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810933    5898 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 04:44:30.822619 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810936    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 04:44:30.822619 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810939    5898 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 04:44:30.822619 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810942    5898 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 04:44:30.823186 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810944    5898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 04:44:30.823186 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810947    5898 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 04:44:30.823186 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810949    5898 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 04:44:30.823186 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810952    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 04:44:30.823186 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810955    5898 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 04:44:30.823186 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810957    5898 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 04:44:30.823186 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810960    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 04:44:30.823186 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810963    5898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 04:44:30.823186 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810966    5898 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 04:44:30.823186 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.810969    5898 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 04:44:30.823186 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.810994    5898 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 04:44:30.823186 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811029    5898 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 04:44:30.823186 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811033    5898 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 04:44:30.823186 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811036    5898 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 04:44:30.823186 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811039    5898 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 04:44:30.823756 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811041    5898 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 04:44:30.823756 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811044    5898 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 04:44:30.823756 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811046    5898 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 04:44:30.823756 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811049    5898 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 04:44:30.823756 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811052    5898 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 04:44:30.823756 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811056    5898 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 04:44:30.823756 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811059    5898 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 04:44:30.823756 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811061    5898 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 04:44:30.823756 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811064    5898 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 04:44:30.823756 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811067    5898 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 04:44:30.823756 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811069    5898 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 04:44:30.823756 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811074    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 04:44:30.823756 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811076    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 04:44:30.823756 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811079    5898 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 04:44:30.823756 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811082    5898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 04:44:30.823756 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811084    5898 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 04:44:30.823756 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811087    5898 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 04:44:30.823756 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811090    5898 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 04:44:30.823756 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811092    5898 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 04:44:30.823756 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811095    5898 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 04:44:30.824741 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811098    5898 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 04:44:30.824741 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811100    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 04:44:30.824741 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811105    5898 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 04:44:30.824741 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811108    5898 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 04:44:30.824741 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811112    5898 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 04:44:30.824741 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811115    5898 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 04:44:30.824741 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811118    5898 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 04:44:30.824741 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811121    5898 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 04:44:30.824741 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811124    5898 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 04:44:30.824741 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811126    5898 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 04:44:30.824741 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811129    5898 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 04:44:30.824741 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811131    5898 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 04:44:30.824741 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811134    5898 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 04:44:30.824741 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811136    5898 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 04:44:30.824741 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811139    5898 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 04:44:30.824741 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811141    5898 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 04:44:30.824741 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811144    5898 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 04:44:30.824741 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811148    5898 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 04:44:30.824741 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811151    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 04:44:30.824741 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811154    5898 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 04:44:30.825477 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811157    5898 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 04:44:30.825477 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811159    5898 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 04:44:30.825477 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811162    5898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 04:44:30.825477 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811166    5898 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 04:44:30.825477 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811168    5898 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 04:44:30.825477 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811171    5898 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 04:44:30.825477 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811174    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 04:44:30.825477 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811176    5898 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 04:44:30.825477 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811179    5898 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 04:44:30.825477 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811182    5898 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 04:44:30.825477 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811184    5898 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 04:44:30.825477 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811187    5898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 04:44:30.825477 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811190    5898 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 04:44:30.825477 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811193    5898 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 04:44:30.825477 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811195    5898 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 04:44:30.825477 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811198    5898 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 04:44:30.825477 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811202    5898 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 04:44:30.825477 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811206    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 04:44:30.825477 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811209    5898 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 04:44:30.826090 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811212    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 04:44:30.826090 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811214    5898 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 04:44:30.826090 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811217    5898 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 04:44:30.826090 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811220    5898 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 04:44:30.826090 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811223    5898 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 04:44:30.826090 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811225    5898 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 04:44:30.826090 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811228    5898 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 04:44:30.826090 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811230    5898 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 04:44:30.826090 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811233    5898 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 04:44:30.826090 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811235    5898 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 04:44:30.826090 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811240    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 04:44:30.826090 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811242    5898 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 04:44:30.826090 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811245    5898 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 04:44:30.826090 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811248    5898 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 04:44:30.826090 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811250    5898 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 04:44:30.826090 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811253    5898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 04:44:30.826090 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811257    5898 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 04:44:30.826090 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811260    5898 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 04:44:30.826090 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811262    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 04:44:30.826090 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811265    5898 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 04:44:30.826675 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811268    5898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 04:44:30.826675 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811271    5898 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 04:44:30.826675 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811274    5898 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 04:44:30.826675 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811276    5898 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 04:44:30.826675 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811279    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 04:44:30.826675 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811282    5898 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 04:44:30.826675 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811284    5898 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 04:44:30.826675 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811287    5898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 04:44:30.826675 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811289    5898 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 04:44:30.826675 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811292    5898 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 04:44:30.826675 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811295    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 04:44:30.826675 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811297    5898 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 04:44:30.826675 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811300    5898 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 04:44:30.826675 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811303    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 04:44:30.826675 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811306    5898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 04:44:30.826675 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811308    5898 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 04:44:30.826675 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811311    5898 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 04:44:30.826675 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811314    5898 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 04:44:30.826675 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811317    5898 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 04:44:30.827166 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811319    5898 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 04:44:30.827166 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811322    5898 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 04:44:30.827166 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811325    5898 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 04:44:30.827166 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.811328    5898 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 04:44:30.827166 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.815688    5898 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 04:44:30.827166 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.815704    5898 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 04:44:30.827166 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.815723    5898 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 04:44:30.827166 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.815727    5898 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 04:44:30.827166 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.815795    5898 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 04:44:30.827166 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.815801    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 04:44:30.827166 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.815805    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 04:44:30.827166 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.815808    5898 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 04:44:30.827166 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.815811    5898 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 04:44:30.827166 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.815814    5898 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 04:44:30.827166 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.815818    5898 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 04:44:30.827166 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.815822    5898 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 04:44:30.827166 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.815825    5898 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 04:44:30.827166 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.815828    5898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 04:44:30.827609 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.815830    5898 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 04:44:30.827609 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.815833    5898 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 04:44:30.827609 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.815836    5898 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 04:44:30.827609 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.815839    5898 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 04:44:30.827609 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.815842    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 04:44:30.827609 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.815844    5898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 04:44:30.827609 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.815847    5898 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 04:44:30.827609 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.815850    5898 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 04:44:30.827609 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.815853    5898 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 04:44:30.827609 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.815856    5898 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 04:44:30.827609 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.815858    5898 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 04:44:30.827609 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.815861    5898 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 04:44:30.827609 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.815865    5898 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 04:44:30.827609 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.815872    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 04:44:30.827609 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.815875    5898 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 04:44:30.827609 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.815878    5898 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 04:44:30.827609 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.815881    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 04:44:30.827609 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.815884    5898 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 04:44:30.827609 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.815887    5898 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 04:44:30.827609 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.815889    5898 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 04:44:30.828163 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.815892    5898 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 04:44:30.828163 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.815895    5898 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 04:44:30.828163 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.815918    5898 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 04:44:30.828163 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.815921    5898 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 04:44:30.828163 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.815924    5898 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 04:44:30.828163 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.815927    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 04:44:30.828163 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.815937    5898 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 04:44:30.828163 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.815940    5898 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 04:44:30.828163 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.815942    5898 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 04:44:30.828163 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.815945    5898 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 04:44:30.828163 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.815948    5898 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 04:44:30.828163 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.815950    5898 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 04:44:30.828163 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.815953    5898 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 04:44:30.828163 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.815955    5898 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 04:44:30.828163 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.815958    5898 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 04:44:30.828163 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.815962    5898 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 04:44:30.828163 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.815964    5898 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 04:44:30.828163 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.815967    5898 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 04:44:30.828163 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.815970    5898 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 04:44:30.828163 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.815972    5898 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 04:44:30.828643 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.815975    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 04:44:30.828643 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.815978    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 04:44:30.828643 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.815980    5898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 04:44:30.828643 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.815984    5898 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 04:44:30.828643 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.815986    5898 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 04:44:30.828643 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.815990    5898 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 04:44:30.828643 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.815999    5898 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 04:44:30.828643 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816003    5898 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 04:44:30.828643 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816006    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 04:44:30.828643 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816008    5898 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 04:44:30.828643 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816011    5898 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 04:44:30.828643 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816014    5898 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 04:44:30.828643 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816016    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 04:44:30.828643 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816019    5898 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 04:44:30.828643 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816021    5898 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 04:44:30.828643 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816024    5898 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 04:44:30.828643 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816027    5898 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 04:44:30.828643 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816030    5898 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 04:44:30.828643 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816033    5898 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 04:44:30.829126 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816035    5898 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 04:44:30.829126 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816038    5898 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 04:44:30.829126 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816040    5898 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 04:44:30.829126 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816043    5898 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 04:44:30.829126 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816046    5898 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 04:44:30.829126 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816049    5898 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 04:44:30.829126 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816051    5898 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 04:44:30.829126 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816054    5898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 04:44:30.829126 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816056    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 04:44:30.829126 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816059    5898 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 04:44:30.829126 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816075    5898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 04:44:30.829126 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816079    5898 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 04:44:30.829126 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816082    5898 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 04:44:30.829126 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816085    5898 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 04:44:30.829126 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816088    5898 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 04:44:30.829126 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816091    5898 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 04:44:30.829126 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816093    5898 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 04:44:30.829126 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816096    5898 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 04:44:30.829126 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816099    5898 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 04:44:30.829601 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816102    5898 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 04:44:30.829601 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816109    5898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 04:44:30.829601 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816112    5898 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 04:44:30.829601 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816115    5898 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 04:44:30.829601 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816117    5898 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 04:44:30.829601 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816120    5898 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 04:44:30.829601 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816122    5898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 04:44:30.829601 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816125    5898 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 04:44:30.829601 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816128    5898 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 04:44:30.829601 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816130    5898 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 04:44:30.829601 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816133    5898 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 04:44:30.829601 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816136    5898 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 04:44:30.829601 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816138    5898 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 04:44:30.829601 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816141    5898 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 04:44:30.829601 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816144    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 04:44:30.829601 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816146    5898 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 04:44:30.829601 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816149    5898 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 04:44:30.829601 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816151    5898 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 04:44:30.830133 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.816179    5898 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 04:44:30.830133 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816367    5898 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 04:44:30.830133 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816371    5898 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 04:44:30.830133 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816374    5898 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 04:44:30.830133 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816377    5898 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 04:44:30.830133 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816380    5898 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 04:44:30.830133 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816384    5898 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 04:44:30.830133 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816388    5898 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 04:44:30.830133 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816390    5898 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 04:44:30.830133 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816393    5898 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 04:44:30.830133 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816396    5898 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 04:44:30.830133 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816399    5898 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 04:44:30.830133 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816402    5898 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 04:44:30.830133 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816404    5898 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 04:44:30.830133 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816407    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 04:44:30.830531 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816409    5898 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 04:44:30.830531 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816416    5898 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 04:44:30.830531 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816419    5898 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 04:44:30.830531 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816422    5898 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 04:44:30.830531 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816425    5898 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 04:44:30.830531 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816427    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 04:44:30.830531 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816430    5898 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 04:44:30.830531 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816433    5898 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 04:44:30.830531 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816435    5898 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 04:44:30.830531 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816438    5898 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 04:44:30.830531 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816441    5898 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 04:44:30.830531 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816443    5898 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 04:44:30.830531 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816446    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 04:44:30.830531 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816448    5898 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 04:44:30.830531 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816451    5898 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 04:44:30.830531 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816454    5898 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 04:44:30.830531 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816456    5898 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 04:44:30.830531 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816458    5898 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 04:44:30.830531 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816461    5898 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 04:44:30.830531 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816463    5898 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 04:44:30.831065 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816466    5898 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 04:44:30.831065 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816468    5898 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 04:44:30.831065 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816471    5898 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 04:44:30.831065 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816474    5898 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 04:44:30.831065 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816476    5898 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 04:44:30.831065 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816480    5898 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 04:44:30.831065 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816484    5898 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 04:44:30.831065 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816486    5898 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 04:44:30.831065 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816489    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 04:44:30.831065 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816492    5898 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 04:44:30.831065 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816496    5898 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 04:44:30.831065 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816499    5898 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 04:44:30.831065 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816502    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 04:44:30.831065 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816505    5898 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 04:44:30.831065 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816512    5898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 04:44:30.831065 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816515    5898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 04:44:30.831065 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816517    5898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 04:44:30.831065 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816520    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 04:44:30.831065 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816522    5898 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 04:44:30.831559 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816525    5898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 04:44:30.831559 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816528    5898 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 04:44:30.831559 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816530    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 04:44:30.831559 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816533    5898 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 04:44:30.831559 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816535    5898 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 04:44:30.831559 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816538    5898 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 04:44:30.831559 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816540    5898 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 04:44:30.831559 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816543    5898 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 04:44:30.831559 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816545    5898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 04:44:30.831559 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816548    5898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 04:44:30.831559 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816550    5898 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 04:44:30.831559 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816553    5898 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 04:44:30.831559 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816555    5898 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 04:44:30.831559 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816558    5898 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 04:44:30.831559 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816560    5898 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 04:44:30.831559 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816563    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 04:44:30.831559 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816565    5898 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 04:44:30.831559 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816568    5898 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 04:44:30.831559 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816570    5898 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 04:44:30.832233 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816573    5898 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 04:44:30.832233 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816575    5898 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 04:44:30.832233 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816578    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 04:44:30.832233 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816581    5898 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 04:44:30.832233 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816583    5898 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 04:44:30.832233 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816586    5898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 04:44:30.832233 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816588    5898 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 04:44:30.832233 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816591    5898 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 04:44:30.832233 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816593    5898 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 04:44:30.832233 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816600    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 04:44:30.832233 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816603    5898 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 04:44:30.832233 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816606    5898 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 04:44:30.832233 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816608    5898 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 04:44:30.832233 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816610    5898 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 04:44:30.832233 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816613    5898 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 04:44:30.832233 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816616    5898 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 04:44:30.832233 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816618    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 04:44:30.832233 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816621    5898 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 04:44:30.832233 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816623    5898 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 04:44:30.832233 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816626    5898 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 04:44:30.832849 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816628    5898 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 04:44:30.832849 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816631    5898 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 04:44:30.832849 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816633    5898 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 04:44:30.832849 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816636    5898 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 04:44:30.832849 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816638    5898 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 04:44:30.832849 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816641    5898 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 04:44:30.832849 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816643    5898 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 04:44:30.832849 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816646    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 04:44:30.832849 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816648    5898 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 04:44:30.832849 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816651    5898 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 04:44:30.832849 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816653    5898 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 04:44:30.832849 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816656    5898 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 04:44:30.832849 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816658    5898 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 04:44:30.832849 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816661    5898 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 04:44:30.832849 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.816684    5898 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 04:44:30.832849 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816721    5898 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 04:44:30.832849 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816724    5898 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 04:44:30.833433 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816727    5898 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 04:44:30.833433 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816730    5898 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 04:44:30.833433 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816733    5898 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 04:44:30.833433 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816736    5898 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 04:44:30.833433 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816738    5898 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 04:44:30.833433 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816745    5898 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 04:44:30.833433 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816748    5898 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 04:44:30.833433 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816751    5898 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 04:44:30.833433 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816754    5898 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 04:44:30.833433 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816757    5898 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 04:44:30.833433 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816761    5898 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 04:44:30.833433 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816764    5898 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 04:44:30.833433 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816767    5898 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 04:44:30.833433 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816769    5898 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 04:44:30.833433 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816772    5898 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 04:44:30.833433 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816774    5898 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 04:44:30.833433 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816777    5898 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 04:44:30.833433 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816780    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 04:44:30.833433 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816782    5898 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 04:44:30.834100 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816785    5898 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 04:44:30.834100 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816787    5898 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 04:44:30.834100 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816790    5898 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 04:44:30.834100 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816792    5898 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 04:44:30.834100 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816795    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 04:44:30.834100 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816798    5898 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 04:44:30.834100 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816800    5898 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 04:44:30.834100 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816803    5898 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 04:44:30.834100 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816805    5898 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 04:44:30.834100 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816808    5898 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 04:44:30.834100 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816811    5898 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 04:44:30.834100 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816813    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 04:44:30.834100 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816816    5898 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 04:44:30.834100 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816818    5898 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 04:44:30.834100 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816821    5898 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 04:44:30.834100 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816823    5898 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 04:44:30.834100 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816826    5898 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 04:44:30.834100 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816828    5898 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 04:44:30.834100 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816831    5898 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 04:44:30.834100 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816839    5898 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 04:44:30.834656 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816842    5898 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 04:44:30.834656 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816845    5898 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 04:44:30.834656 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816847    5898 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 04:44:30.834656 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816850    5898 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 04:44:30.834656 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816853    5898 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 04:44:30.834656 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816858    5898 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 04:44:30.834656 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.816969    5898 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 04:44:30.834656 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.817524    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 04:44:30.834656 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.817532    5898 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 04:44:30.834656 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.817537    5898 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 04:44:30.834656 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.817542    5898 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 04:44:30.834656 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.817552    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 04:44:30.834656 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.817557    5898 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 04:44:30.834656 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.817561    5898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 04:44:30.834656 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.817567    5898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 04:44:30.834656 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.817571    5898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 04:44:30.834656 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.817576    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 04:44:30.834656 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.817580    5898 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 04:44:30.834656 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.817591    5898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 04:44:30.835194 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.817596    5898 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 04:44:30.835194 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.817600    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 04:44:30.835194 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.817605    5898 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 04:44:30.835194 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.817609    5898 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 04:44:30.835194 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.817613    5898 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 04:44:30.835194 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.817619    5898 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 04:44:30.835194 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.817623    5898 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 04:44:30.835194 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.817628    5898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 04:44:30.835194 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.817632    5898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 04:44:30.835194 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.817637    5898 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 04:44:30.835194 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.817641    5898 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 04:44:30.835194 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.817645    5898 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 04:44:30.835194 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.817654    5898 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 04:44:30.835194 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.817659    5898 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 04:44:30.835194 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.817663    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 04:44:30.835194 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.817667    5898 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 04:44:30.835194 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.817672    5898 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 04:44:30.835194 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.817676    5898 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 04:44:30.835194 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.817681    5898 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 04:44:30.836054 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.817685    5898 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 04:44:30.836054 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.817695    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 04:44:30.836054 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.817700    5898 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 04:44:30.836054 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.817705    5898 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 04:44:30.836054 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.817709    5898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 04:44:30.836054 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.817713    5898 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 04:44:30.836054 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.817723    5898 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 04:44:30.836054 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.817727    5898 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 04:44:30.836054 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.817731    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 04:44:30.836054 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.817735    5898 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 04:44:30.836054 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.817739    5898 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 04:44:30.836054 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.817744    5898 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 04:44:30.836054 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.817748    5898 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 04:44:30.836054 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.817753    5898 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 04:44:30.836054 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.817757    5898 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 04:44:30.836054 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.817761    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 04:44:30.836054 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.817765    5898 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 04:44:30.836054 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.817769    5898 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 04:44:30.836054 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.817778    5898 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 04:44:30.836054 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.817783    5898 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 04:44:30.836872 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.817790    5898 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 04:44:30.836872 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.817795    5898 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 04:44:30.836872 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.817800    5898 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 04:44:30.836872 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.817804    5898 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 04:44:30.836872 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.817808    5898 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 04:44:30.836872 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.817812    5898 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 04:44:30.836872 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:30.817817    5898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 04:44:30.836872 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.817950    5898 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 04:44:30.836872 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.819602    5898 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 04:44:30.836872 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.819670    5898 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 04:44:30.836872 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.819960    5898 server.go:1014] "Starting client certificate rotation"
Oct 30 04:44:30.836872 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.820082    5898 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 04:44:30.836872 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.820131    5898 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 04:38:19 +0000 UTC" deadline="2028-06-16 11:47:36.090725074 +0000 UTC"
Oct 30 04:44:30.836872 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.820158    5898 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14287h3m5.27056921s"
Oct 30 04:44:30.836872 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.820600    5898 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 04:44:30.836872 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.821840    5898 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 04:44:30.837313 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.823003    5898 log.go:25] "Validated CRI v1 runtime API"
Oct 30 04:44:30.837313 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.825398    5898 log.go:25] "Validated CRI v1 image API"
Oct 30 04:44:30.837313 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.826164    5898 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 04:44:30.837313 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.828165    5898 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 b2f2c13c-3afe-4217-bacb-b7de744e0a49:/dev/nvme0n1p3 ecc471dc-8048-4b0a-82b0-9f40149b53b3:/dev/nvme0n1p4]
Oct 30 04:44:30.837470 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.828184    5898 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/03cda3c4c09016f9eed112c104471848e90fd8df45e68b3d01027033385a64bf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/03cda3c4c09016f9eed112c104471848e90fd8df45e68b3d01027033385a64bf/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/06617a34cae62d587573f703a0f9751f896e6ffbd092ce68c4cb339ca96b4c6a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/06617a34cae62d587573f703a0f9751f896e6ffbd092ce68c4cb339ca96b4c6a/userdata/shm major:0 minor:368 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1c2c7e85e42f36c1dd3132a5fc87e0b98b6d22ea9dec0a9c5cd39844c3050fa9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1c2c7e85e42f36c1dd3132a5fc87e0b98b6d22ea9dec0a9c5cd39844c3050fa9/userdata/shm major:0 minor:299 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1f57f7ee285d7f2006cb35fd89c62f6d4029fb5a9c788ddc354a196be0931cd4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1f57f7ee285d7f2006cb35fd89c62f6d4029fb5a9c788ddc354a196be0931cd4/userdata/shm major:0 minor:292 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2337701261d2bc376ff506ce1806be3ee7b49efec319b3e1b59e756b19f4f424/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2337701261d2bc376ff506ce1806be3ee7b49efec319b3e1b59e756b19f4f424/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2e751d48664d410a227b1bec20c5227213cafb620e21361743051bb9854998f7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2e751d48664d410a227b1bec20c5227213cafb620e21361743051bb9854998f7/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/52237db8a2daea675ec54c056e14360e3eced11310ea7b7a06f9771cbc8fa81d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/52237db8a2daea675ec54c056e14360e3eced11310ea7b7a06f9771cbc8fa81d/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/55e9f36fd3d87ecd971b3336a06458ceae8c710341869364e8912f232c844aac/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/55e9f36fd3d87ecd971b3336a06458ceae8c710341869364e8912f232c844aac/userdata/shm major:0 minor:355 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/622b5622e07128e7fe97898d1e42bf52701f806897f41e5c1599820de4a78858/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/622b5622e07128e7fe97898d1e42bf52701f806897f41e5c1599820de4a78858/userdata/shm major:0 minor:100 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6653524753f250fb105e5601c4942691e41dc39723b6868b480553e6e4fd3d16/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6653524753f250fb105e5601c4942691e41dc39723b6868b480553e6e4fd3d16/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/744019019c1faa264b370e28aa0390f184677b13e9690c64273d2b0a8e103943/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/744019019c1faa264b370e28aa0390f184677b13e9690c64273d2b0a8e103943/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7b4b0fdfa213c0d6369f78967e3e637a533d14c38e5ea2c175afde0913212db3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7b4b0fdfa213c0d6369f78967e3e637a533d14c38e5ea2c175afde0913212db3/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/83f871ee138c5e31cdb59e3f27b898b80540b16fb8ac9bad0f2056868d7dc4f4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/83f871ee138c5e31cdb59e3f27b898b80540b16fb8ac9bad0f2056868d7dc4f4/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9b517736b4c15deb64649ad76e845dfc28dd4505bb717cdb56f37e5f9e23363a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9b517736b4c15deb64649ad76e845dfc28dd4505bb717cdb56f37e5f9e23363a/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9ed996658a95d487489d306eef164e7090b0c202b685ef01b49e6ac9d768be91/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9ed996658a95d487489d306eef164e7090b0c202b685ef01b49e6ac9d768be91/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ae34671c26d15cc2d08cba5274e6c4ba4ebae35967ce18ff62ae9a627b41cfd5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ae34671c26d15cc2d08cba5274e6c4ba4ebae35967ce18ff62ae9a627b41cfd5/userdata/shm major:0 minor:370 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cecccb7ad501a7ae51a7855e1a1426bcbdc1141471d05d904280a45ff1211b96/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cecccb7ad501a7ae51a7855e1a1426bcbdc1141471d05d904280a45ff1211b96/userdata/shm major:0 minor:271 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e25a610fa35edc5ec7910b2cd0e9454bdbd9dd7942289d7bfea79b931518573b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e25a610fa35edc5ec7910b2cd0e9454bdbd9dd7942289d7bfea79b931518573b/userdata/shm major:0 minor:371 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/efb8094f5548de960a06cbfadeed8eb0424196b2bcdd35525ef699ec6ff36aa4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/efb8094f5548de960a06cbfadeed8eb0424196b2bcdd35525ef699ec6ff36aa4/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f7cadb24483929c3c6476697bf27ec2f6bcb8b095253c00d16ea594659edc7da/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f7cadb24483929c3c6476697bf27ec2f6bcb8b095253c00d16ea594659edc7da/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fc25014589e21d741b04fdcef4af9d6eec45f39f3afd74e3b0e3f95b25e58d35/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fc25014589e21d741b04fdcef4af9d6eec45f39f3afd74e3b0e3f95b25e58d35/userdata/shm major:0 minor:83 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fdbac03ac556c63dcaef04cfa683bab4791e457aaa8db342b5538ceea9d38aba/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fdbac03ac556c63dcaef04cfa683bab4791e457aaa8db342b5538ceea9d38aba/userdata/shm major:0 minor:305 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3023960e-7dab-48fe-926b-b016f2e2b5b7/volumes/kubernetes.io~projected/kube-api-access-mhgw5:{mountpoint:/var/lib/kubelet/pods/3023960e-7dab-48fe-926b-b016f2e2b5b7/volumes/kubernetes.io~projected/kube-api-access-mhgw5 major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3023960e-7dab-48fe-926b-b016f2e2b5b7/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/3023960e-7dab-48fe-926b-b016f2e2b5b7/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3023960e-7dab-48fe-926b-b016f2e2b5b7/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/3023960e-7dab-48fe-926b-b016f2e2b5b7/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b1cae8a-12cf-4b59-85f0-a63f324fc8e1/volumes/kubernetes.io~projected/kube-api-access-f8x7k:{mountpoint:/var/lib/kubelet/pods/3b1cae8a-12cf-4b59-85f0-a63f324fc8e1/volumes/kubernetes.io~projected/kube-api-access-f8x7k major:0 minor:367 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/41a4ec4b-b407-4316-9238-dee6be432d3c/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/41a4ec4b-b407-4316-9238-dee6be432d3c/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:361 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/47f03fac-07ba-4d50-9336-c5af8b4fbcc3/volumes/kubernetes.io~projected/kube-api-access-z6rb2:{mountpoint:/var/lib/kubelet/pods/47f03fac-07ba-4d50-9336-c5af8b4fbcc3/volumes/kubernetes.io~projected/kube-api-access-z6rb2 major:0 minor:354 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/47f03fac-07ba-4d50-9336-c5af8b4fbcc3/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/47f03fac-07ba-4d50-9336-c5af8b4fbcc3/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:348 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/47f03fac-07ba-4d50-9336-c5af8b4fbcc3/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/47f03fac-07ba-4d50-9336-c5af8b4fbcc3/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:353 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/672524df-2f52-455b-9423-a58f53c31129/volumes/kubernetes.io~projected/kube-api-access-bfwq8:{mountpoint:/var/lib/kubelet/pods/672524df-2f52-455b-9423-a58f53c31129/volumes/kubernetes.io~projected/kube-api-access-bfwq8 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70169887-cc72-4fb5-96b5-e09e260d7a8b/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/70169887-cc72-4fb5-96b5-e09e260d7a8b/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70169887-cc72-4fb5-96b5-e09e260d7a8b/volumes/kubernetes.io~projected/kube-api-access-pknnw:{mountpoint:/var/lib/kubelet/pods/70169887-cc72-4fb5-96b5-e09e260d7a8b/volumes/kubernetes.io~projected/kube-api-access-pknnw major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70169887-cc72-4fb5-96b5-e09e260d7a8b/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/70169887-cc72-4fb5-96b5-e09e260d7a8b/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73a78a0d-d19e-4a96-b8ce-253d631da703/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/73a78a0d-d19e-4a96-b8ce-253d631da703/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/76081e78-ba20-4f7c-b825-a9e0da23c511/volumes/kubernetes.io~projected/kube-api-access-4dg2h:{mountpoint:/var/lib/kubelet/pods/76081e78-ba20-4f7c-b825-a9e0da23c511/volumes/kubernetes.io~projected/kube-api-access-4dg2h major:0 minor:270 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/76081e78-ba20-4f7c-b825-a9e0da23c511/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/76081e78-ba20-4f7c-b825-a9e0da23c511/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:269 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/76081e78-ba20-4f7c-b825-a9e0da23c511/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/76081e78-ba20-4f7c-b825-a9e0da23c511/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:265 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7ad09519-e98b-4efc-8b6b-ec7d031c3c2f/volumes/kubernetes.io~projected/kube-api-access-qld8j:{mountpoint:/var/lib/kubelet/pods/7ad09519-e98b-4efc-8b6b-ec7d031c3c2f/volumes/kubernetes.io~projected/kube-api-access-qld8j major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7ad09519-e98b-4efc-8b6b-ec7d031c3c2f/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/7ad09519-e98b-4efc-8b6b-ec7d031c3c2f/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7f75d879-5e31-41fd-838c-b1f0408cb45a/volumes/kubernetes.io~projected/kube-api-access-49nsv:{mountpoint:/var/lib/kubelet/pods/7f75d879-5e31-41fd-838c-b1f0408cb45a/volumes/kubernetes.io~projected/kube-api-access-49nsv major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7f75d879-5e31-41fd-838c-b1f0408cb45a/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/7f75d879-5e31-41fd-838c-b1f0408cb45a/volumes/kubernetes.io~secret/metrics-certs major:0 minor:366 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/885d476c-7568-4b63-ab9d-1da7608706f5/volumes/kubernetes.io~projected/kube-api-access-kwcw4:{mountpoint:/var/lib/kubelet/pods/885d476c-7568-4b63-ab9d-1da7608706f5/volumes/kubernetes.io~projected/kube-api-access-kwcw4 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a2a3eb4b-a128-4649-bf77-78df2705f147/volumes/kubernetes.io~projected/kube-api-access-nxpck:{mountpoint:/var/lib/kubelet/pods/a2a3eb4b-a128-4649-bf77-78df2705f147/volumes/kubernetes.io~projected/kube-api-access-nxpck major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf49335f-7b8e-4212-8777-1575cd0929f1/volumes/kubernetes.io~projected/kube-api-access-4w8n4:{mountpoint:/var/lib/kubelet/pods/bf49335f-7b8e-4212-8777-1575cd0929f1/volumes/kubernetes.io~projected/kube-api-access-4w8n4 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bff48d36-3909-4838-b499-66e5081bcc90/volumes/kubernetes.io~projected/kube-api-access-l294n:{mountpoint:/var/lib/kubelet/pods/bff48d36-3909-4838-b499-66e5081bcc90/volumes/kubernetes.io~projected/kube-api-access-l294n major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d/volumes/kubernetes.io~projected/kube-api-access-gv4sg:{mountpoint:/var/lib/kubelet/pods/d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d/volumes/kubernetes.io~projected/kube-api-access-gv4sg major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/daf6535c-7f33-407a-a5d9-e1c9525c8cb8/volumes/kubernetes.io~projected/kube-api-access-29k58:{mountpoint:/var/lib/kubelet/pods/daf6535c-7f33-407a-a5d9-e1c9525c8cb8/volumes/kubernetes.io~projected/kube-api-access-29k58 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ddad100f-6a08-4da8-a51a-f3f2fd5438bb/volumes/kubernetes.io~projected/kube-api-access-vmbs5:{mountpoint:/var/lib/kubelet/pods/ddad100f-6a08-4da8-a51a-f3f2fd5438bb/volumes/kubernetes.io~projected/kube-api-access-vmbs5 major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ddad100f-6a08-4da8-a51a-f3f2fd5438bb/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/ddad100f-6a08-4da8-a51a-f3f2fd5438bb/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eddb252a-1213-4eb9-9219-5a0abedf5587/volumes/kubernetes.io~projected/kube-api-access-klznl:{mountpoint:/var/lib/kubelet/pods/eddb252a-1213-4eb9-9219-5a0abedf5587/volumes/kubernetes.io~projected/kube-api-access-klznl major:0 minor:304 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eddb252a-1213-4eb9-9219-5a0abedf5587/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/eddb252a-1213-4eb9-9219-5a0abedf5587/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:303 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/faa6a40d-2e9e-4348-a599-0df6f2d0177d/volumes/kubernetes.io~projected/kube-api-access-p5q8q:{mountpoint:/var/lib/kubelet/pods/faa6a40d-2e9e-4348-a599-0df6f2d0177d/volumes/kubernetes.io~projected/kube-api-access-p5q8q major:0 minor:298 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/faa6a40d-2e9e-4348-a599-0df6f2d0177d/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/faa6a40d-2e9e-4348-a599-0df6f2d0177d/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:297 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/faa6a40d-2e9e-4348-a599-0df6f2d0177d/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/faa6a40d-2e9e-4348-a599-0df6f2d0177d/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fd380d55-276e-45eb-8191-c5940ed8385b/volumes/kubernetes.io~projected/kube-api-access-t7tw7:{mountpoint:/var/lib/kubelet/pods/fd380d55-276e-45eb-8191-c5940ed8385b/volumes/kubernetes.io~projected/kube-api-access-t7tw7 major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fd380d55-276e-45eb-8191-c5940ed8385b/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/fd380d55-276e-45eb-8191-c5940ed8385b/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fd380d55-276e-45eb-8191-c5940ed8385b/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/fd380d55-276e-45eb-8191-c5940ed8385b/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fd380d55-276e-45eb-8191-c5940ed8385b/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/fd380d55-276e-45eb-8191-c5940ed8385b/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fd380d55-276e-45eb-8191-c5940ed8385b/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/fd380d55-276e-45eb-8191-c5940ed8385b/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fd380d55-276e-45eb-8191-c5940ed8385b/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/fd380d55-276e-45eb-8191-c5940ed8385b/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fd380d55-276e-45eb-8191-c5940ed8385b/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/fd380d55-276e-45eb-8191-c5940ed8385b/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935/volumes/kubernetes.io~projected/kube-api-access-92qd4:{mountpoint:/var/lib/kubelet/pods/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935/volumes/kubernetes.io~projected/kube-api-access-92qd4 major:0 minor:63 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/becec028e4065770124ddc248acaff8e00d156e86fd7b890efd82b4d52929a50/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/953f40dd7ee3a7ccee3fa0df70aa2920863afe515b56c6dbabfb94621276befc/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/b80f131cf9d19113568882d5780ac6ac4bc5ddb195d1774826c168db07a5fd6c/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/7065a7ee276c0751af02d4a20aac97e96950a0d7c7427f2c630409b5f80a923c/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/b5986445a1b59fc9d71639ceeb1c79fb31b9025a43f15032fae23a2d014a7d59/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/4047803c27bf23730f3d6b7f297069b84ec0fc59528da7123eadc57935d75cc3/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/dd6382eaaf0dae0cfaa3d202b8e82b6bf2f8ab24cc0bb4a08eed4c3c363b814e/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/49bc65b56203c5306259152e4b44a5bd552375c7ca6767969eea59c89056ce8c/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/3a1b78587a97e852c9fe0132e898a7581fc5fd7f02b98115696c8a420f78419e/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-136:{mountpoint:/var/lib/containers/storage/overlay/2d568e91e6c3b1842146a5b8264adfaccfe3e7e00c6c16e20382b8cf8050298a/merged major:0 minor:136 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/containers/storage/overlay/62404495b31600493f4d9a4ad99c173aa4097506e7016ad62d6d1e46c6721b90/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/2d7c4220fccc1fa756ee7d4d8b3a90a12ef683c4e63d9114e5fffe2d14920e41/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/889084bba20af0ff3fd91235ee280d709d48f0024367e4fe0fe34d8674a5b393/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/41c6849ee4008df3aecf692adf7b42400de728e5331247c30c1b39b9d8e6f9e5/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/3b895bd31aa2a42d3c9b099f03ce4b9609b1b387aebab49d36c1b33acc3f9aec/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/b5e42c71c72162eeb3f94510326be9af1aafc790c082fdd9730c86a96d7f5a20/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/590c91217780a69cf8560ff9368a0d671d2da90a15233250699b10524b20a158/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/9d4687136ad9a1dfbd73d9208cb20e8011b3a74773c5ac3ab9faa118f510b402/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/4f9830ffa563435cdcbdca7f13776c63fda4ff66229e1dbed0bae3be919ff661/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/e511dccbec844129f74ffc74728377d46602831eb6a1b1ba1a1faac5dff9d87e/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/ddb6ee9d5d560b6383fba2880699f99fe4318858037bd423f1787bec80e914cc/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/b95c9cd157d3aa85a79b3c93a21dc8b7fc5dc5fcca19829586a916692149e552/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/15678cdb17e4bab0517024f7b18a061ca807fcdacbcddbf2aba4ad16d901531e/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/5a1d4d4f783ff69654a19907315c0983425928af2818c12f319461430e989e9c/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/8667b5822dc2b22d2a602cec18082be847af4e868bdd27b06316f0d55e59eb23/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/e991a376e2be4dbbded9d6cee7499f2f081550d20343a5ea9e89cce4d366e110/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/fce614becfc1ebf39dbbcb1c89978efba2ba978f95d77dbb1ff1f12cd15e1198/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/4672580d0a877d5f59de68f7d4776a1d3132d4966afe1b0ad67950e01597eab1/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-243:{mountpoint:/var/lib/containers/storage/overlay/0ff7922dc38b9e753a74b5607baf5229881be724d917fb621bd7005350cab7cd/merged major:0 minor:243 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/5cb5a47a94d926bf9fb352c4c032dbfaec2dd4d5fcabe89971a53d62c5390a4b/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/295542d6e6e928f280ea572b5c27a1dd85a3269b56b752d13d3fe3425e6e4f7d/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/245fd92625cc8beeed915abbe92c7a2d1f7f1e9923f28f3e603562b2d7312706/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/3135e356e27a4788a7f1257507528a3d4a6490beb3375c16114cd38918c8adac/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-275:{mountpoint:/var/lib/containers/storage/overlay/c907174ce675ff05a93c22ef1ad1417c5c89cf5968d6d30431527c58419b4cd6/merged major:0 minor:275 fsType:overlay blockSize:0} overlay_0-277:{mountpoint:/var/lib/containers/storage/overlay/8f686fc6595197620650398b9ffdb0fb1af4c701d953463118d65b0792ab31ca/merged major:0 minor:277 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/180d8a683b4178d877b9af592314e21916af8bf95364c45ab30d34af31f93417/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-301:{mountpoint:/var/lib/containers/storage/overlay/305ae7d7de39a10aafb03d19cc772ac6f62236d7d53f785fc54aed19f534ee39/merged major:0 minor:301 fsType:overlay blockSize:0} overlay_0-307:{mountpoint:/var/lib/containers/storage/overlay/77d6d1da30d6d14b846c491e21f7de9055ea9b2b7ab911d521bb6a089aa0e603/merged major:0 minor:307 fsType:overlay blockSize:0} overlay_0-309:{mountpoint:/var/lib/containers/storage/overlay/ecca328749b8c994261ac203dd2a50227103036a49312ea1a6cf5da3287db0c3/merged major:0 minor:309 fsType:overlay blockSize:0} overlay_0-315:{mountpoint:/var/lib/containers/storage/overlay/057288063b8b3a5f1bb71c854538cfc2d17fdda5cb37f97d81109f299df34712/merged major:0 minor:315 fsType:overlay blockSize:0} overlay_0-320:{mountpoint:/var/lib/containers/storage/overlay/5d5970b95e032620dc90ec22ddc2bd7753ced08d080bcbc72c8c7dd31c9c0394/merged major:0 minor:320 fsType:overlay blockSize:0} overlay_0-322:{mountpoint:/var/lib/containers/storage/overlay/2f1d69f67d8db4be40600369ca9513fdb1cbf2359e51aab51ff7d13170f87739/merged major:0 minor:322 fsType:overlay blockSize:0} overlay_0-327:{mountpoint:/var/lib/containers/storage/overlay/6ed31c43e3c5122f8d925d08d98d34d91a84565e4d2ecdf93df102c3c54a3a8a/merged major:0 minor:327 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/595e8f6f62988378ac5ff4fd07752b4a6108e0a9c4714a69d85db5471c8b6599/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-337:{mountpoint:/var/lib/containers/storage/overlay/f22135702199d7e794f3fee75f7f673a5d556817d0191087fffca6e6cc8c3835/merged major:0 minor:337 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/041fe1d43a590def30e74aeb56d938e3673384dc4d748a27e64b75a33d1686a1/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-357:{mountpoint:/var/lib/containers/storage/overlay/295a828140f8997ae28813dc3ea4061eea55017403f16cc911a42bd62605201b/merged major:0 minor:357 fsType:overlay blockSize:0} overlay_0-359:{mountpoint:/var/lib/containers/storage/overlay/22c2dba20c353d1db2af6b6badca3df893daf051815b86e414374dfc62e570ab/merged major:0 minor:359 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/55bc62e433dfa489a30b2009ac5c7ebc9a462f5c1045900220b9e784c2623d5c/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/a1a90a0c5816135db8c820fd620d18f4038173e354b5920f8b4a6e7e02bfd459/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-378:{mountpoint:/var/lib/containers/storage/overlay/d7a25131f2e3575fb5c254898d4dd716edfc9fe0165c0aedb8e79ef1e21931ae/merged major:0 minor:378 fsType:overlay blockSize:0} overlay_0-380:{mountpoint:/var/lib/containers/storage/overlay/3ce64e4a605585d21356746646d373b32ca2cefbc905e0822d2ec6e65491ac88/merged major:0 minor:380 fsType:overlay blockSize:0} overlay_0-382:{mountpoint:/var/lib/containers/storage/overlay/75571c67f61be195dfe6c56bbc20b58a471752fd5637ffaaebb1026c60fc7871/merged major:0 minor:382 fsType:overlay blockSize:0} overlay_0-388:{mountpoint:/var/lib/containers/storage/overlay/d0d338022e2f6ea7b37d1b33ba5f50c028e1acaa05aa8be680bfe51f65582254/merged major:0 minor:388 fsType:overlay blockSize:0} overlay_0-393:{mountpoint:/var/lib/containers/storage/overlay/fabba7e6a30925569c8f4e39e6c77a4cdffe753d07d347977cb1d29f8ef00749/merged major:0 minor:393 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/d179c5a6e5573f353a7fef4bdad304b988fd47c3782f4c91709677d9edcfc562/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/864c2c13b2f1d13b043270c207dfd7cadeb1919a118cf47c3f723b3aabb85573/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/05938cbb54e113b94e511a939c0455a47e94b666ffb5c989c1d757917ac4de3a/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/1056d9fa3a881de7a4c7ccf6691eb3daabcbcf76f6385c6fb7482709b89c550d/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/774e2f91fccab50bb4c99e9100585d8b7d2e17058755860ff09d9e7ef0a9a19e/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/0240c7285831cf2db8696697a72f972051e4c9863e851c4d877bdda1ddda51de/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-85:{mountpoint:/var/lib/containers/storage/overlay/661e40db1082b04bf98f88de26653907460cf1871fee8a6a2961eb232e9a55ce/merged major:0 minor:85 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/e13bbba2ee2d2fa614ab244d00a0957ab9a0745fa10b7611d9b5537c733af51e/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/8e406dc68588ec3887d6ddbb84400517b4f1876fe84e225365833868d1751741/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/e6ec952441a6a1358374252610e9892548f6efebfe415d51dd7d58999effa286/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/54193caceaa63678ea1b71c8b294b7b730353b21bb43e4ea158b5378a74a2de1/merged major:0 minor:98 fsType:overlay blockSize:0}]
Oct 30 04:44:30.838051 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.837130    5898 manager.go:217] Machine: {Timestamp:2026-10-30 04:44:30.83609808 +0000 UTC m=+0.059917974 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100230 MemoryCapacity:33164324864 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21f93a24d2f5567a3b9cb0461d9fd6 SystemUUID:ec21f93a-24d2-f556-7a3b-9cb0461d9fd6 BootID:7b07d1f0-e52b-4d8c-a7f1-03975f01a029 Filesystems:[{Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-85 DeviceMajor:0 DeviceMinor:85 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7ad09519-e98b-4efc-8b6b-ec7d031c3c2f/volumes/kubernetes.io~projected/kube-api-access-qld8j DeviceMajor:0 DeviceMinor:204 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f7cadb24483929c3c6476697bf27ec2f6bcb8b095253c00d16ea594659edc7da/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ddad100f-6a08-4da8-a51a-f3f2fd5438bb/volumes/kubernetes.io~projected/kube-api-access-vmbs5 DeviceMajor:0 DeviceMinor:203 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7ad09519-e98b-4efc-8b6b-ec7d031c3c2f/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/7f75d879-5e31-41fd-838c-b1f0408cb45a/volumes/kubernetes.io~projected/kube-api-access-49nsv DeviceMajor:0 DeviceMinor:61 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/622b5622e07128e7fe97898d1e42bf52701f806897f41e5c1599820de4a78858/userdata/shm DeviceMajor:0 DeviceMinor:100 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-388 DeviceMajor:0 DeviceMinor:388 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3023960e-7dab-48fe-926b-b016f2e2b5b7/volumes/kubernetes.io~projected/kube-api-access-mhgw5 DeviceMajor:0 DeviceMinor:230 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/fd380d55-276e-45eb-8191-c5940ed8385b/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:286 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-359 DeviceMajor:0 DeviceMinor:359 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/70169887-cc72-4fb5-96b5-e09e260d7a8b/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:201 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-337 DeviceMajor:0 DeviceMinor:337 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/41a4ec4b-b407-4316-9238-dee6be432d3c/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:361 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/52237db8a2daea675ec54c056e14360e3eced11310ea7b7a06f9771cbc8fa81d/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/76081e78-ba20-4f7c-b825-a9e0da23c511/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:269 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-315 DeviceMajor:0 DeviceMinor:315 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935/volumes/kubernetes.io~projected/kube-api-access-92qd4 DeviceMajor:0 DeviceMinor:63 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3023960e-7dab-48fe-926b-b016f2e2b5b7/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:225 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/47f03fac-07ba-4d50-9336-c5af8b4fbcc3/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:353 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/744019019c1faa264b370e28aa0390f184677b13e9690c64273d2b0a8e103943/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/fd380d55-276e-45eb-8191-c5940ed8385b/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:290 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582164480 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/03cda3c4c09016f9eed112c104471848e90fd8df45e68b3d01027033385a64bf/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-141 DeviceMajor:0 DeviceMinor:141 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/70169887-cc72-4fb5-96b5-e09e260d7a8b/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/fd380d55-276e-45eb-8191-c5940ed8385b/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:287 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1c2c7e85e42f36c1dd3132a5fc87e0b98b6d22ea9dec0a9c5cd39844c3050fa9/userdata/shm DeviceMajor:0 DeviceMinor:299 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/faa6a40d-2e9e-4348-a599-0df6f2d0177d/volumes/kubernetes.io~projected/kube-api-access-p5q8q DeviceMajor:0 DeviceMinor:298 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-322 DeviceMajor:0 DeviceMinor:322 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3b1cae8a-12cf-4b59-85f0-a63f324fc8e1/volumes/kubernetes.io~projected/kube-api-access-f8x7k DeviceMajor:0 DeviceMinor:367 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/70169887-cc72-4fb5-96b5-e09e260d7a8b/volumes/kubernetes.io~projected/kube-api-access-pknnw DeviceMajor:0 DeviceMinor:65 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/faa6a40d-2e9e-4348-a599-0df6f2d0177d/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:296 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-382 DeviceMajor:0 DeviceMinor:382 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9b517736b4c15deb64649ad76e845dfc28dd4505bb717cdb56f37e5f9e23363a/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/885d476c-7568-4b63-ab9d-1da7608706f5/volumes/kubernetes.io~projected/kube-api-access-kwcw4 DeviceMajor:0 DeviceMinor:64 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/55e9f36fd3d87ecd971b3336a06458ceae8c710341869364e8912f232c844aac/userdata/shm DeviceMajor:0 DeviceMinor:355 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/a2a3eb4b-a128-4649-bf77-78df2705f147/volumes/kubernetes.io~projected/kube-api-access-nxpck DeviceMajor:0 DeviceMinor:60 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-275 DeviceMajor:0 DeviceMinor:275 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-380 DeviceMajor:0 DeviceMinor:380 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fd380d55-276e-45eb-8191-c5940ed8385b/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:288 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2e751d48664d410a227b1bec20c5227213cafb620e21361743051bb9854998f7/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9ed996658a95d487489d306eef164e7090b0c202b685ef01b49e6ac9d768be91/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/daf6535c-7f33-407a-a5d9-e1c9525c8cb8/volumes/kubernetes.io~projected/kube-api-access-29k58 DeviceMajor:0 DeviceMinor:67 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-309 DeviceMajor:0 DeviceMinor:309 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-393 DeviceMajor:0 DeviceMinor:393 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bff48d36-3909-4838-b499-66e5081bcc90/volumes/kubernetes.io~projected/kube-api-access-l294n DeviceMajor:0 DeviceMinor:66 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d/volumes/kubernetes.io~projected/kube-api-access-gv4sg DeviceMajor:0 DeviceMinor:202 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/76081e78-ba20-4f7c-b825-a9e0da23c511/volumes/kubernetes.io~projected/kube-api-access-4dg2h DeviceMajor:0 DeviceMinor:270 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-277 DeviceMajor:0 DeviceMinor:277 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-320 DeviceMajor:0 DeviceMinor:320 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7f75d879-5e31-41fd-838c-b1f0408cb45a/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:366 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-378 DeviceMajor:0 DeviceMinor:378 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2337701261d2bc376ff506ce1806be3ee7b49efec319b3e1b59e756b19f4f424/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e25a610fa35edc5ec7910b2cd0e9454bdbd9dd7942289d7bfea79b931518573b/userdata/shm DeviceMajor:0 DeviceMinor:371 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-243 DeviceMajor:0 DeviceMinor:243 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fdbac03ac556c63dcaef04cfa683bab4791e457aaa8db342b5538ceea9d38aba/userdata/shm DeviceMajor:0 DeviceMinor:305 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-136 DeviceMajor:0 DeviceMinor:136 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cecccb7ad501a7ae51a7855e1a1426bcbdc1141471d05d904280a45ff1211b96/userdata/shm DeviceMajor:0 DeviceMinor:271 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/fd380d55-276e-45eb-8191-c5940ed8385b/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:289 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-327 DeviceMajor:0 DeviceMinor:327 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/47f03fac-07ba-4d50-9336-c5af8b4fbcc3/volumes/kubernetes.io~projected/kube-api-access-z6rb2 DeviceMajor:0 DeviceMinor:354 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3023960e-7dab-48fe-926b-b016f2e2b5b7/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:229 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/76081e78-ba20-4f7c-b825-a9e0da23c511/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:265 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/faa6a40d-2e9e-4348-a599-0df6f2d0177d/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:297 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/73a78a0d-d19e-4a96-b8ce-253d631da703/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fc25014589e21d741b04fdcef4af9d6eec45f39f3afd74e3b0e3f95b25e58d35/userdata/shm DeviceMajor:0 DeviceMinor:83 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582160384 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/bf49335f-7b8e-4212-8777-1575cd0929f1/volumes/kubernetes.io~projected/kube-api-access-4w8n4 DeviceMajor:0 DeviceMinor:62 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6653524753f250fb105e5601c4942691e41dc39723b6868b480553e6e4fd3d16/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/06617a34cae62d587573f703a0f9751f896e6ffbd092ce68c4cb339ca96b4c6a/userdata/shm DeviceMajor:0 DeviceMinor:368 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7b4b0fdfa213c0d6369f78967e3e637a533d14c38e5ea2c175afde0913212db3/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1f57f7ee285d7f2006cb35fd89c62f6d4029fb5a9c788ddc354a196be0931cd4/userdata/shm DeviceMajor:0 DeviceMinor:292 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/eddb252a-1213-4eb9-9219-5a0abedf5587/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:303 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-357 DeviceMajor:0 DeviceMinor:357 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-307 DeviceMajor:0 DeviceMinor:307 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ae34671c26d15cc2d08cba5274e6c4ba4ebae35967ce18ff62ae9a627b41cfd5/userdata/shm DeviceMajor:0 DeviceMinor:370 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/672524df-2f52-455b-9423-a58f53c31129/volumes/kubernetes.io~projected/kube-api-access-bfwq8 DeviceMajor:0 DeviceMinor:59 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/47f03fac-07ba-4d50-9336-c5af8b4fbcc3/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:348 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fd380d55-276e-45eb-8191-c5940ed8385b/volumes/kubernetes.io~projected/kube-api-access-t7tw7 DeviceMajor:0 DeviceMinor:291 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/eddb252a-1213-4eb9-9219-5a0abedf5587/volumes/kubernetes.io~projected/kube-api-access-klznl DeviceMajor:0 DeviceMinor:304 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/fd380d55-276e-45eb-8191-c5940ed8385b/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:282 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/ddad100f-6a08-4da8-a51a-f3f2fd5438bb/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/efb8094f5548de960a06cbfadeed8eb0424196b2bcdd35525ef699ec6ff36aa4/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/83f871ee138c5e31cdb59e3f27b898b80540b16fb8ac9bad0f2056868d7dc4f4/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-301 DeviceMajor:0 DeviceMinor:301 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:06617a34cae62d5 MacAddress:32:0a:3c:17:e7:14 Speed:10000 Mtu:8901} {Name:1c2c7e85e42f36c MacAddress:56:0c:7b:e2:38:5a Speed:10000 Mtu:8901} {Name:1f57f7ee285d7f2 MacAddress:a6:97:cd:b0:28:01 Speed:10000 Mtu:8901} {Name:55e9f36fd3d87ec MacAddress:e2:21:c0:14:fd:0f Speed:10000 Mtu:8901} {Name:6653524753f250f MacAddress:26:3f:4e:6d:48:85 Speed:10000 Mtu:8901} {Name:744019019c1faa2 MacAddress:26:4b:55:9c:83:53 Speed:10000 Mtu:8901} {Name:83f871ee138c5e3 MacAddress:5a:9f:4b:f1:fa:b7 Speed:10000 Mtu:8901} {Name:ae34671c26d15cc MacAddress:3a:36:c2:18:1c:7c Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:dc:18:ca:55 Speed:0 Mtu:9001} {Name:br-int MacAddress:de:4f:a0:e5:d1:87 Speed:0 Mtu:8901} {Name:e25a610fa35edc5 MacAddress:72:e2:70:21:dc:7e Speed:10000 Mtu:8901} {Name:efb8094f5548de9 MacAddress:a2:c6:60:46:d7:48 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:dc:18:ca:55 Speed:0 Mtu:9001} {Name:fdbac03ac556c63 MacAddress:72:33:1f:36:13:42 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:5a:d0:0b:f8:80:bd Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:c2:12:8d:11:55:db Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164324864 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 04:44:30.838051 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.837513    5898 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 04:44:30.838051 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.837560    5898 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 04:44:30.838051 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.837842    5898 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 04:44:30.838051 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.837865    5898 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-98.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 04:44:30.838051 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.838008    5898 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 04:44:30.838051 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.838016    5898 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 04:44:30.838051 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.838027    5898 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 04:44:30.838051 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.838045    5898 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 04:44:30.838480 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.838128    5898 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 04:44:30.838480 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.838292    5898 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 04:44:30.838480 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.838406    5898 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 04:44:30.838480 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.838440    5898 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 04:44:30.838480 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.838453    5898 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 04:44:30.838480 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.838463    5898 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 04:44:30.838480 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.838472    5898 kubelet.go:405] "Adding apiserver pod source"
Oct 30 04:44:30.838480 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.838480    5898 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 04:44:30.839226 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.839210    5898 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 04:44:30.839226 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.839227    5898 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 04:44:30.839729 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.839710    5898 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 04:44:30.843103 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.843081    5898 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 04:44:30.843103 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.843101    5898 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 04:44:30.843215 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.843205    5898 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 04:44:30.843400 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.843389    5898 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 04:44:30.843440 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.843418    5898 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 04:44:30.843557 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.843545    5898 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 04:44:30.843592 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.843565    5898 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 04:44:30.843592 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.843572    5898 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 04:44:30.843592 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.843578    5898 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 04:44:30.843592 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.843584    5898 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 04:44:30.843592 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.843590    5898 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 04:44:30.843727 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.843595    5898 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 04:44:30.843727 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.843601    5898 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 04:44:30.843727 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.843608    5898 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 04:44:30.843727 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.843614    5898 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 04:44:30.843727 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.843623    5898 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 04:44:30.843727 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.843633    5898 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 04:44:30.843727 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.843665    5898 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 04:44:30.843727 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.843674    5898 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 04:44:30.847169 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.847152    5898 server.go:1263] "Started kubelet"
Oct 30 04:44:30.847312 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.847249    5898 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 04:44:30.847399 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.847376    5898 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 04:44:30.847966 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.847943    5898 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 04:44:30.848080 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.847929    5898 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 04:44:30.848390 ip-10-0-135-98 systemd[1]: Started Kubernetes Kubelet.
Oct 30 04:44:30.849879 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.849863    5898 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 04:44:30.855700 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.854233    5898 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 04:44:30.855700 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.854423    5898 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 04:44:30.855700 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.854527    5898 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 04:38:21 +0000 UTC" deadline="2028-06-09 16:24:03.27999303 +0000 UTC"
Oct 30 04:44:30.855700 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.854575    5898 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14123h39m32.425421922s"
Oct 30 04:44:30.855700 ip-10-0-135-98 kubenswrapper[5898]: E1030 04:44:30.854733    5898 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-98.ec2.internal\" not found"
Oct 30 04:44:30.855700 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.854778    5898 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 04:44:30.855700 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.854790    5898 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 04:44:30.855700 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.854916    5898 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 04:44:30.858601 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.858571    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="faa6a40d-2e9e-4348-a599-0df6f2d0177d" volumeName="kubernetes.io/secret/faa6a40d-2e9e-4348-a599-0df6f2d0177d-console-oauth-config" seLinuxMountContext=""
Oct 30 04:44:30.858707 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.858602    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="885d476c-7568-4b63-ab9d-1da7608706f5" volumeName="kubernetes.io/projected/885d476c-7568-4b63-ab9d-1da7608706f5-kube-api-access-kwcw4" seLinuxMountContext=""
Oct 30 04:44:30.858707 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.858618    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ddad100f-6a08-4da8-a51a-f3f2fd5438bb" volumeName="kubernetes.io/configmap/ddad100f-6a08-4da8-a51a-f3f2fd5438bb-config-volume" seLinuxMountContext=""
Oct 30 04:44:30.858707 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.858631    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ddad100f-6a08-4da8-a51a-f3f2fd5438bb" volumeName="kubernetes.io/empty-dir/ddad100f-6a08-4da8-a51a-f3f2fd5438bb-tmp-dir" seLinuxMountContext=""
Oct 30 04:44:30.858707 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.858643    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd380d55-276e-45eb-8191-c5940ed8385b" volumeName="kubernetes.io/secret/fd380d55-276e-45eb-8191-c5940ed8385b-secret-thanos-querier-tls" seLinuxMountContext=""
Oct 30 04:44:30.858707 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.858654    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47f03fac-07ba-4d50-9336-c5af8b4fbcc3" volumeName="kubernetes.io/configmap/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-oauth-serving-cert" seLinuxMountContext=""
Oct 30 04:44:30.858707 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.858665    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70169887-cc72-4fb5-96b5-e09e260d7a8b" volumeName="kubernetes.io/configmap/70169887-cc72-4fb5-96b5-e09e260d7a8b-ovnkube-config" seLinuxMountContext=""
Oct 30 04:44:30.858707 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.858677    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f75d879-5e31-41fd-838c-b1f0408cb45a" volumeName="kubernetes.io/secret/7f75d879-5e31-41fd-838c-b1f0408cb45a-metrics-certs" seLinuxMountContext=""
Oct 30 04:44:30.858707 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.858691    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd380d55-276e-45eb-8191-c5940ed8385b" volumeName="kubernetes.io/secret/fd380d55-276e-45eb-8191-c5940ed8385b-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext=""
Oct 30 04:44:30.858707 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.858701    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76081e78-ba20-4f7c-b825-a9e0da23c511" volumeName="kubernetes.io/secret/76081e78-ba20-4f7c-b825-a9e0da23c511-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 04:44:30.858707 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.858712    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47f03fac-07ba-4d50-9336-c5af8b4fbcc3" volumeName="kubernetes.io/secret/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-console-serving-cert" seLinuxMountContext=""
Oct 30 04:44:30.859238 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.858723    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd380d55-276e-45eb-8191-c5940ed8385b" volumeName="kubernetes.io/configmap/fd380d55-276e-45eb-8191-c5940ed8385b-metrics-client-ca" seLinuxMountContext=""
Oct 30 04:44:30.859238 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.858734    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd380d55-276e-45eb-8191-c5940ed8385b" volumeName="kubernetes.io/secret/fd380d55-276e-45eb-8191-c5940ed8385b-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext=""
Oct 30 04:44:30.859238 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.858746    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2a3eb4b-a128-4649-bf77-78df2705f147" volumeName="kubernetes.io/configmap/a2a3eb4b-a128-4649-bf77-78df2705f147-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 04:44:30.859238 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.858760    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf49335f-7b8e-4212-8777-1575cd0929f1" volumeName="kubernetes.io/projected/bf49335f-7b8e-4212-8777-1575cd0929f1-kube-api-access-4w8n4" seLinuxMountContext=""
Oct 30 04:44:30.859238 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.858768    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d" volumeName="kubernetes.io/configmap/d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 30 04:44:30.859238 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.858777    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="faa6a40d-2e9e-4348-a599-0df6f2d0177d" volumeName="kubernetes.io/configmap/faa6a40d-2e9e-4348-a599-0df6f2d0177d-oauth-serving-cert" seLinuxMountContext=""
Oct 30 04:44:30.859238 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.858789    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47f03fac-07ba-4d50-9336-c5af8b4fbcc3" volumeName="kubernetes.io/configmap/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-service-ca" seLinuxMountContext=""
Oct 30 04:44:30.859238 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.858800    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="daf6535c-7f33-407a-a5d9-e1c9525c8cb8" volumeName="kubernetes.io/configmap/daf6535c-7f33-407a-a5d9-e1c9525c8cb8-serviceca" seLinuxMountContext=""
Oct 30 04:44:30.859238 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.858810    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ddad100f-6a08-4da8-a51a-f3f2fd5438bb" volumeName="kubernetes.io/projected/ddad100f-6a08-4da8-a51a-f3f2fd5438bb-kube-api-access-vmbs5" seLinuxMountContext=""
Oct 30 04:44:30.859238 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.858819    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd380d55-276e-45eb-8191-c5940ed8385b" volumeName="kubernetes.io/secret/fd380d55-276e-45eb-8191-c5940ed8385b-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext=""
Oct 30 04:44:30.859238 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.858829    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70169887-cc72-4fb5-96b5-e09e260d7a8b" volumeName="kubernetes.io/configmap/70169887-cc72-4fb5-96b5-e09e260d7a8b-env-overrides" seLinuxMountContext=""
Oct 30 04:44:30.859238 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.858837    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3023960e-7dab-48fe-926b-b016f2e2b5b7" volumeName="kubernetes.io/projected/3023960e-7dab-48fe-926b-b016f2e2b5b7-kube-api-access-mhgw5" seLinuxMountContext=""
Oct 30 04:44:30.859238 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.858846    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ad09519-e98b-4efc-8b6b-ec7d031c3c2f" volumeName="kubernetes.io/secret/7ad09519-e98b-4efc-8b6b-ec7d031c3c2f-cert" seLinuxMountContext=""
Oct 30 04:44:30.859238 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.858857    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2a3eb4b-a128-4649-bf77-78df2705f147" volumeName="kubernetes.io/configmap/a2a3eb4b-a128-4649-bf77-78df2705f147-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 04:44:30.859238 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.858868    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bff48d36-3909-4838-b499-66e5081bcc90" volumeName="kubernetes.io/projected/bff48d36-3909-4838-b499-66e5081bcc90-kube-api-access-l294n" seLinuxMountContext=""
Oct 30 04:44:30.859238 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.858878    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff3a5b12-bdd2-47d1-9475-ee3b2bcba935" volumeName="kubernetes.io/empty-dir/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-etc-tuned" seLinuxMountContext=""
Oct 30 04:44:30.859238 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.858890    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3023960e-7dab-48fe-926b-b016f2e2b5b7" volumeName="kubernetes.io/secret/3023960e-7dab-48fe-926b-b016f2e2b5b7-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 04:44:30.859238 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.858973    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47f03fac-07ba-4d50-9336-c5af8b4fbcc3" volumeName="kubernetes.io/configmap/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 04:44:30.859238 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.858986    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47f03fac-07ba-4d50-9336-c5af8b4fbcc3" volumeName="kubernetes.io/secret/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-console-oauth-config" seLinuxMountContext=""
Oct 30 04:44:30.859238 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.858998    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76081e78-ba20-4f7c-b825-a9e0da23c511" volumeName="kubernetes.io/empty-dir/76081e78-ba20-4f7c-b825-a9e0da23c511-node-exporter-textfile" seLinuxMountContext=""
Oct 30 04:44:30.859238 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.859009    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ad09519-e98b-4efc-8b6b-ec7d031c3c2f" volumeName="kubernetes.io/projected/7ad09519-e98b-4efc-8b6b-ec7d031c3c2f-kube-api-access-qld8j" seLinuxMountContext=""
Oct 30 04:44:30.859238 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.859017    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f75d879-5e31-41fd-838c-b1f0408cb45a" volumeName="kubernetes.io/projected/7f75d879-5e31-41fd-838c-b1f0408cb45a-kube-api-access-49nsv" seLinuxMountContext=""
Oct 30 04:44:30.859238 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.859024    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff3a5b12-bdd2-47d1-9475-ee3b2bcba935" volumeName="kubernetes.io/empty-dir/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-tmp" seLinuxMountContext=""
Oct 30 04:44:30.859238 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.859031    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b1cae8a-12cf-4b59-85f0-a63f324fc8e1" volumeName="kubernetes.io/projected/3b1cae8a-12cf-4b59-85f0-a63f324fc8e1-kube-api-access-f8x7k" seLinuxMountContext=""
Oct 30 04:44:30.859238 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.859039    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47f03fac-07ba-4d50-9336-c5af8b4fbcc3" volumeName="kubernetes.io/empty-dir/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-tmp" seLinuxMountContext=""
Oct 30 04:44:30.859238 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.859055    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="885d476c-7568-4b63-ab9d-1da7608706f5" volumeName="kubernetes.io/empty-dir/885d476c-7568-4b63-ab9d-1da7608706f5-tmp-dir" seLinuxMountContext=""
Oct 30 04:44:30.859238 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.859063    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d" volumeName="kubernetes.io/empty-dir/d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d-data-volume" seLinuxMountContext=""
Oct 30 04:44:30.859238 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.859071    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ddad100f-6a08-4da8-a51a-f3f2fd5438bb" volumeName="kubernetes.io/secret/ddad100f-6a08-4da8-a51a-f3f2fd5438bb-metrics-tls" seLinuxMountContext=""
Oct 30 04:44:30.859238 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.859078    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76081e78-ba20-4f7c-b825-a9e0da23c511" volumeName="kubernetes.io/projected/76081e78-ba20-4f7c-b825-a9e0da23c511-kube-api-access-4dg2h" seLinuxMountContext=""
Oct 30 04:44:30.859238 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.859086    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76081e78-ba20-4f7c-b825-a9e0da23c511" volumeName="kubernetes.io/secret/76081e78-ba20-4f7c-b825-a9e0da23c511-node-exporter-tls" seLinuxMountContext=""
Oct 30 04:44:30.859238 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.859094    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bff48d36-3909-4838-b499-66e5081bcc90" volumeName="kubernetes.io/configmap/bff48d36-3909-4838-b499-66e5081bcc90-iptables-alerter-script" seLinuxMountContext=""
Oct 30 04:44:30.859238 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.859101    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="faa6a40d-2e9e-4348-a599-0df6f2d0177d" volumeName="kubernetes.io/projected/faa6a40d-2e9e-4348-a599-0df6f2d0177d-kube-api-access-p5q8q" seLinuxMountContext=""
Oct 30 04:44:30.859238 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.859109    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd380d55-276e-45eb-8191-c5940ed8385b" volumeName="kubernetes.io/secret/fd380d55-276e-45eb-8191-c5940ed8385b-secret-grpc-tls" seLinuxMountContext=""
Oct 30 04:44:30.859238 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.859117    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3023960e-7dab-48fe-926b-b016f2e2b5b7" volumeName="kubernetes.io/configmap/3023960e-7dab-48fe-926b-b016f2e2b5b7-metrics-client-ca" seLinuxMountContext=""
Oct 30 04:44:30.859238 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.859125    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41a4ec4b-b407-4316-9238-dee6be432d3c" volumeName="kubernetes.io/secret/41a4ec4b-b407-4316-9238-dee6be432d3c-original-pull-secret" seLinuxMountContext=""
Oct 30 04:44:30.859238 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.859132    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="672524df-2f52-455b-9423-a58f53c31129" volumeName="kubernetes.io/projected/672524df-2f52-455b-9423-a58f53c31129-kube-api-access-bfwq8" seLinuxMountContext=""
Oct 30 04:44:30.859238 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.859140    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70169887-cc72-4fb5-96b5-e09e260d7a8b" volumeName="kubernetes.io/projected/70169887-cc72-4fb5-96b5-e09e260d7a8b-kube-api-access-pknnw" seLinuxMountContext=""
Oct 30 04:44:30.859238 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.859147    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76081e78-ba20-4f7c-b825-a9e0da23c511" volumeName="kubernetes.io/configmap/76081e78-ba20-4f7c-b825-a9e0da23c511-metrics-client-ca" seLinuxMountContext=""
Oct 30 04:44:30.859238 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.859156    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2a3eb4b-a128-4649-bf77-78df2705f147" volumeName="kubernetes.io/configmap/a2a3eb4b-a128-4649-bf77-78df2705f147-cni-binary-copy" seLinuxMountContext=""
Oct 30 04:44:30.859238 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.859163    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d" volumeName="kubernetes.io/secret/d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d-insights-runtime-extractor-tls" seLinuxMountContext=""
Oct 30 04:44:30.859238 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.859177    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="faa6a40d-2e9e-4348-a599-0df6f2d0177d" volumeName="kubernetes.io/configmap/faa6a40d-2e9e-4348-a599-0df6f2d0177d-console-config" seLinuxMountContext=""
Oct 30 04:44:30.859238 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.859185    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73a78a0d-d19e-4a96-b8ce-253d631da703" volumeName="kubernetes.io/configmap/73a78a0d-d19e-4a96-b8ce-253d631da703-konnectivity-ca" seLinuxMountContext=""
Oct 30 04:44:30.859238 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.859192    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="daf6535c-7f33-407a-a5d9-e1c9525c8cb8" volumeName="kubernetes.io/projected/daf6535c-7f33-407a-a5d9-e1c9525c8cb8-kube-api-access-29k58" seLinuxMountContext=""
Oct 30 04:44:30.859238 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.859199    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eddb252a-1213-4eb9-9219-5a0abedf5587" volumeName="kubernetes.io/secret/eddb252a-1213-4eb9-9219-5a0abedf5587-monitoring-plugin-cert" seLinuxMountContext=""
Oct 30 04:44:30.859238 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.859207    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="faa6a40d-2e9e-4348-a599-0df6f2d0177d" volumeName="kubernetes.io/secret/faa6a40d-2e9e-4348-a599-0df6f2d0177d-console-serving-cert" seLinuxMountContext=""
Oct 30 04:44:30.859238 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.859220    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd380d55-276e-45eb-8191-c5940ed8385b" volumeName="kubernetes.io/projected/fd380d55-276e-45eb-8191-c5940ed8385b-kube-api-access-t7tw7" seLinuxMountContext=""
Oct 30 04:44:30.859238 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.859228    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd380d55-276e-45eb-8191-c5940ed8385b" volumeName="kubernetes.io/secret/fd380d55-276e-45eb-8191-c5940ed8385b-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 30 04:44:30.859238 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.859235    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47f03fac-07ba-4d50-9336-c5af8b4fbcc3" volumeName="kubernetes.io/configmap/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-console-config" seLinuxMountContext=""
Oct 30 04:44:30.859238 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.859243    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47f03fac-07ba-4d50-9336-c5af8b4fbcc3" volumeName="kubernetes.io/projected/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-kube-api-access-z6rb2" seLinuxMountContext=""
Oct 30 04:44:30.859238 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.859250    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76081e78-ba20-4f7c-b825-a9e0da23c511" volumeName="kubernetes.io/configmap/76081e78-ba20-4f7c-b825-a9e0da23c511-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Oct 30 04:44:30.859238 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.859258    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d" volumeName="kubernetes.io/projected/d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d-kube-api-access-gv4sg" seLinuxMountContext=""
Oct 30 04:44:30.859238 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.859265    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eddb252a-1213-4eb9-9219-5a0abedf5587" volumeName="kubernetes.io/projected/eddb252a-1213-4eb9-9219-5a0abedf5587-kube-api-access-klznl" seLinuxMountContext=""
Oct 30 04:44:30.859238 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.859273    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="faa6a40d-2e9e-4348-a599-0df6f2d0177d" volumeName="kubernetes.io/configmap/faa6a40d-2e9e-4348-a599-0df6f2d0177d-service-ca" seLinuxMountContext=""
Oct 30 04:44:30.861712 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.859282    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd380d55-276e-45eb-8191-c5940ed8385b" volumeName="kubernetes.io/empty-dir/fd380d55-276e-45eb-8191-c5940ed8385b-tmp" seLinuxMountContext=""
Oct 30 04:44:30.861712 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.859294    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff3a5b12-bdd2-47d1-9475-ee3b2bcba935" volumeName="kubernetes.io/projected/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-kube-api-access-92qd4" seLinuxMountContext=""
Oct 30 04:44:30.861712 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.859305    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3023960e-7dab-48fe-926b-b016f2e2b5b7" volumeName="kubernetes.io/secret/3023960e-7dab-48fe-926b-b016f2e2b5b7-prometheus-operator-tls" seLinuxMountContext=""
Oct 30 04:44:30.861712 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.859317    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70169887-cc72-4fb5-96b5-e09e260d7a8b" volumeName="kubernetes.io/configmap/70169887-cc72-4fb5-96b5-e09e260d7a8b-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 04:44:30.861712 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.859324    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73a78a0d-d19e-4a96-b8ce-253d631da703" volumeName="kubernetes.io/secret/73a78a0d-d19e-4a96-b8ce-253d631da703-agent-certs" seLinuxMountContext=""
Oct 30 04:44:30.861712 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.859331    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf49335f-7b8e-4212-8777-1575cd0929f1" volumeName="kubernetes.io/configmap/bf49335f-7b8e-4212-8777-1575cd0929f1-multus-daemon-config" seLinuxMountContext=""
Oct 30 04:44:30.861712 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.859339    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70169887-cc72-4fb5-96b5-e09e260d7a8b" volumeName="kubernetes.io/secret/70169887-cc72-4fb5-96b5-e09e260d7a8b-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 04:44:30.861712 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.859347    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2a3eb4b-a128-4649-bf77-78df2705f147" volumeName="kubernetes.io/projected/a2a3eb4b-a128-4649-bf77-78df2705f147-kube-api-access-nxpck" seLinuxMountContext=""
Oct 30 04:44:30.861712 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.859358    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="faa6a40d-2e9e-4348-a599-0df6f2d0177d" volumeName="kubernetes.io/empty-dir/faa6a40d-2e9e-4348-a599-0df6f2d0177d-tmp" seLinuxMountContext=""
Oct 30 04:44:30.861712 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.859369    5898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf49335f-7b8e-4212-8777-1575cd0929f1" volumeName="kubernetes.io/configmap/bf49335f-7b8e-4212-8777-1575cd0929f1-cni-binary-copy" seLinuxMountContext=""
Oct 30 04:44:30.861712 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.859377    5898 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 04:44:30.861712 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.859383    5898 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 04:44:30.861712 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.859969    5898 factory.go:153] Registering CRI-O factory
Oct 30 04:44:30.861712 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.859990    5898 factory.go:223] Registration of the crio container factory successfully
Oct 30 04:44:30.861712 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.860043    5898 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 04:44:30.861712 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.860052    5898 factory.go:55] Registering systemd factory
Oct 30 04:44:30.861712 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.860058    5898 factory.go:223] Registration of the systemd container factory successfully
Oct 30 04:44:30.861712 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.860069    5898 factory.go:103] Registering Raw factory
Oct 30 04:44:30.861712 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.860079    5898 manager.go:1199] Started watching for new ooms in manager
Oct 30 04:44:30.861712 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.860329    5898 manager.go:319] Starting recovery of all containers
Oct 30 04:44:30.880338 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.880282    5898 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 04:44:30.881965 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.881932    5898 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 04:44:30.881965 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.881953    5898 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 04:44:30.882146 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.881976    5898 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 04:44:30.882146 ip-10-0-135-98 kubenswrapper[5898]: E1030 04:44:30.882019    5898 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 04:44:30.897949 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.897924    5898 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 04:44:30.898686 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.898661    5898 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 04:44:30.906095 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.906060    5898 generic.go:356] "Generic (PLEG): container finished" podID="76081e78-ba20-4f7c-b825-a9e0da23c511" containerID="512c27af9a0f400fe7957e95f53cd03c0306d65e5b9b599ca8e8e1fb62387359" exitCode=0
Oct 30 04:44:30.908639 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.908612    5898 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 04:44:30.920521 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.920496    5898 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 04:44:30.921045 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.921025    5898 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 04:44:30.932369 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.932345    5898 generic.go:356] "Generic (PLEG): container finished" podID="b4286e30fd1fc24b304f69bb80b9a852" containerID="235f96ed71c0e7f50b8cc1411b837d1e00a1fc051da2365bd87ecce415d6f82d" exitCode=0
Oct 30 04:44:30.945528 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.945504    5898 generic.go:356] "Generic (PLEG): container finished" podID="a2a3eb4b-a128-4649-bf77-78df2705f147" containerID="93da0d17151a83471e41b64d2d9cff19580aeaa3e200eb2a4203f5308ceeb9cf" exitCode=0
Oct 30 04:44:30.945528 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.945526    5898 generic.go:356] "Generic (PLEG): container finished" podID="a2a3eb4b-a128-4649-bf77-78df2705f147" containerID="1dfcf90a6c53a224c3570c4940ae9c61cdb35c9d06f67dc06d05ee22f81ab23c" exitCode=0
Oct 30 04:44:30.945641 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.945533    5898 generic.go:356] "Generic (PLEG): container finished" podID="a2a3eb4b-a128-4649-bf77-78df2705f147" containerID="789be87cb8f6d7efccf0b242d6e5e83afbedda95f133412978302941a7957e30" exitCode=0
Oct 30 04:44:30.945641 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.945540    5898 generic.go:356] "Generic (PLEG): container finished" podID="a2a3eb4b-a128-4649-bf77-78df2705f147" containerID="a8f7f10b87ebe77015fdc121beab034f6837531fdcb02d7513d08c0581b11105" exitCode=0
Oct 30 04:44:30.945641 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.945545    5898 generic.go:356] "Generic (PLEG): container finished" podID="a2a3eb4b-a128-4649-bf77-78df2705f147" containerID="70b5a7864be8534112c788ac29f42b3f3974399b71f6b90cd480b7b65466eb8f" exitCode=0
Oct 30 04:44:30.945641 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.945552    5898 generic.go:356] "Generic (PLEG): container finished" podID="a2a3eb4b-a128-4649-bf77-78df2705f147" containerID="887ebee1bedc93413e427e8634657307f14f3ecff030671b8d486d090bb2df2e" exitCode=0
Oct 30 04:44:30.964445 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.964429    5898 manager.go:324] Recovery completed
Oct 30 04:44:30.982152 ip-10-0-135-98 kubenswrapper[5898]: E1030 04:44:30.982098    5898 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 30 04:44:30.984814 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.984795    5898 cpu_manager.go:226] "Starting" policy="none"
Oct 30 04:44:30.984814 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.984810    5898 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 04:44:30.984970 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.984828    5898 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 04:44:30.985032 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.984969    5898 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 04:44:30.985032 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.984979    5898 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 04:44:30.985032 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.984995    5898 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 04:44:30.985032 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.985001    5898 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 04:44:30.985032 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.985006    5898 policy_none.go:50] "Start"
Oct 30 04:44:30.985032 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.985014    5898 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 04:44:30.985032 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.985023    5898 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 04:44:30.985221 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.985170    5898 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 04:44:30.985221 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.985177    5898 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 04:44:30.985221 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.985183    5898 policy_none.go:44] "Start"
Oct 30 04:44:30.989551 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.989526    5898 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 04:44:30.992158 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.992144    5898 manager.go:341] "Starting Device Plugin manager"
Oct 30 04:44:30.992849 ip-10-0-135-98 kubenswrapper[5898]: E1030 04:44:30.992170    5898 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 04:44:30.992894 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.992855    5898 server.go:87] "Starting device plugin registration server"
Oct 30 04:44:30.993210 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.993198    5898 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 04:44:30.993244 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.993213    5898 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 04:44:30.993300 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.993284    5898 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 04:44:30.993408 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.993374    5898 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 04:44:30.993408 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.993410    5898 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 04:44:30.993551 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.993418    5898 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 04:44:30.993551 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.993464    5898 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T04:44:30.993400688Z","UUID":"a4d70bf3-28bd-4366-bddc-1f5356508622","Handler":null,"Name":"","Endpoint":""}
Oct 30 04:44:30.995876 ip-10-0-135-98 kubenswrapper[5898]: E1030 04:44:30.995854    5898 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 04:44:30.996526 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.996493    5898 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 04:44:30.996526 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:30.996523    5898 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 04:44:31.094277 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:31.094252    5898 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 04:44:31.096418 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:31.096395    5898 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-98.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 04:44:31.096525 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:31.096433    5898 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-98.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 04:44:31.096525 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:31.096447    5898 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-98.ec2.internal" event="NodeHasSufficientPID"
Oct 30 04:44:31.096525 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:31.096497    5898 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-98.ec2.internal"
Oct 30 04:44:31.128504 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:31.128476    5898 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-135-98.ec2.internal"
Oct 30 04:44:31.128600 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:31.128547    5898 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-98.ec2.internal"
Oct 30 04:44:31.182343 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:31.182305    5898 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-98.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-98.ec2.internal"]
Oct 30 04:44:31.183057 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:31.183042    5898 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-98.ec2.internal"
Oct 30 04:44:31.183117 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:31.183082    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-98.ec2.internal" event={"ID":"b4286e30fd1fc24b304f69bb80b9a852","Type":"ContainerStarted","Data":"fbc8a8656cf7e6b8c2e962abd1965ed6e175cb717ddb802632f59ecb6abdee2d"}
Oct 30 04:44:31.183153 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:31.183125    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-98.ec2.internal" event={"ID":"b4286e30fd1fc24b304f69bb80b9a852","Type":"ContainerDied","Data":"235f96ed71c0e7f50b8cc1411b837d1e00a1fc051da2365bd87ecce415d6f82d"}
Oct 30 04:44:31.183153 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:31.183147    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-98.ec2.internal" event={"ID":"b4286e30fd1fc24b304f69bb80b9a852","Type":"ContainerStarted","Data":"9b517736b4c15deb64649ad76e845dfc28dd4505bb717cdb56f37e5f9e23363a"}
Oct 30 04:44:31.183207 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:31.183166    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-98.ec2.internal" event={"ID":"38fe9c19c3aaab70dd96caa47ec7e827","Type":"ContainerStarted","Data":"6a8ce66c4e40fcd7b3b1c86c553bfa2fe98099b33796195cd1d8235ac79fee4e"}
Oct 30 04:44:31.183207 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:31.183184    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-98.ec2.internal" event={"ID":"38fe9c19c3aaab70dd96caa47ec7e827","Type":"ContainerStarted","Data":"03cda3c4c09016f9eed112c104471848e90fd8df45e68b3d01027033385a64bf"}
Oct 30 04:44:31.183262 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:31.183216    5898 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-98.ec2.internal"
Oct 30 04:44:31.210554 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:31.210537    5898 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 04:44:31.210616 ip-10-0-135-98 kubenswrapper[5898]: E1030 04:44:31.210573    5898 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-98.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-98.ec2.internal"
Oct 30 04:44:31.210823 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:31.210813    5898 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 04:44:31.210854 ip-10-0-135-98 kubenswrapper[5898]: E1030 04:44:31.210837    5898 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-98.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-98.ec2.internal"
Oct 30 04:44:31.301197 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:31.301139    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/38fe9c19c3aaab70dd96caa47ec7e827-config\") pod \"kube-apiserver-proxy-ip-10-0-135-98.ec2.internal\" (UID: \"38fe9c19c3aaab70dd96caa47ec7e827\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-98.ec2.internal"
Oct 30 04:44:31.301197 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:31.301167    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b4286e30fd1fc24b304f69bb80b9a852-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-98.ec2.internal\" (UID: \"b4286e30fd1fc24b304f69bb80b9a852\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-98.ec2.internal"
Oct 30 04:44:31.301197 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:31.301186    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b4286e30fd1fc24b304f69bb80b9a852-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-98.ec2.internal\" (UID: \"b4286e30fd1fc24b304f69bb80b9a852\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-98.ec2.internal"
Oct 30 04:44:31.401740 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:31.401714    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/38fe9c19c3aaab70dd96caa47ec7e827-config\") pod \"kube-apiserver-proxy-ip-10-0-135-98.ec2.internal\" (UID: \"38fe9c19c3aaab70dd96caa47ec7e827\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-98.ec2.internal"
Oct 30 04:44:31.401845 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:31.401745    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b4286e30fd1fc24b304f69bb80b9a852-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-98.ec2.internal\" (UID: \"b4286e30fd1fc24b304f69bb80b9a852\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-98.ec2.internal"
Oct 30 04:44:31.401845 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:31.401761    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b4286e30fd1fc24b304f69bb80b9a852-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-98.ec2.internal\" (UID: \"b4286e30fd1fc24b304f69bb80b9a852\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-98.ec2.internal"
Oct 30 04:44:31.401845 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:31.401820    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/38fe9c19c3aaab70dd96caa47ec7e827-config\") pod \"kube-apiserver-proxy-ip-10-0-135-98.ec2.internal\" (UID: \"38fe9c19c3aaab70dd96caa47ec7e827\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-98.ec2.internal"
Oct 30 04:44:31.401845 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:31.401832    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b4286e30fd1fc24b304f69bb80b9a852-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-98.ec2.internal\" (UID: \"b4286e30fd1fc24b304f69bb80b9a852\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-98.ec2.internal"
Oct 30 04:44:31.402000 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:31.401820    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b4286e30fd1fc24b304f69bb80b9a852-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-98.ec2.internal\" (UID: \"b4286e30fd1fc24b304f69bb80b9a852\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-98.ec2.internal"
Oct 30 04:44:31.839060 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:31.839028    5898 apiserver.go:52] "Watching apiserver"
Oct 30 04:44:31.879532 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:31.878790    5898 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 04:44:31.880230 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:31.880202    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jghzj","kube-system/kube-apiserver-proxy-ip-10-0-135-98.ec2.internal","openshift-image-registry/node-ca-zqs6v","openshift-console/console-7db77977b9-2f8qv","openshift-network-operator/iptables-alerter-st55p","openshift-multus/network-metrics-daemon-vjfrn","openshift-monitoring/node-exporter-9mr8m","openshift-dns/dns-default-bbk54","openshift-monitoring/thanos-querier-95b457c5b-n92bg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhsc7","openshift-monitoring/prometheus-operator-9cd89b74c-4z6wz","openshift-ovn-kubernetes/ovnkube-node-v4bnv","kube-system/global-pull-secret-syncer-tt5mj","kube-system/konnectivity-agent-6bpl4","openshift-multus/multus-c9tpc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-98.ec2.internal","openshift-cluster-node-tuning-operator/tuned-qtb8v","openshift-console/console-f66c46c7c-skmxs","openshift-dns/node-resolver-gndxs","openshift-multus/multus-additional-cni-plugins-tnwc6","openshift-insights/insights-runtime-extractor-srsjc","openshift-network-diagnostics/network-check-target-dgsdt","openshift-monitoring/monitoring-plugin-77f4f76594-r5tdl"]
Oct 30 04:44:31.899599 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:31.899567    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 04:44:31.899722 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:31.899644    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 04:44:31.899722 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:31.899651    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-45bnw\""
Oct 30 04:44:31.899722 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:31.899644    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 04:44:31.899722 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:31.899658    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 04:44:31.899722 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:31.899685    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 04:44:31.899722 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:31.899735    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 04:44:31.900152 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:31.899811    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-56dd5\""
Oct 30 04:44:31.900152 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:31.899851    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mxnkb\""
Oct 30 04:44:31.900152 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:31.899954    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bpk6m\""
Oct 30 04:44:31.900152 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:31.900021    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 04:44:31.900152 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:31.900092    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 04:44:31.900324 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:31.900182    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 04:44:31.905559 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:31.905543    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 04:44:31.948581 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:31.948557    5898 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-98.ec2.internal"
Oct 30 04:44:31.948718 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:31.948701    5898 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-98.ec2.internal"
Oct 30 04:44:31.956616 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:31.956588    5898 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 04:44:31.970677 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:31.970654    5898 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 04:44:31.970741 ip-10-0-135-98 kubenswrapper[5898]: E1030 04:44:31.970702    5898 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-98.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-98.ec2.internal"
Oct 30 04:44:31.971242 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:31.971224    5898 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 04:44:31.971300 ip-10-0-135-98 kubenswrapper[5898]: E1030 04:44:31.971271    5898 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-98.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-98.ec2.internal"
Oct 30 04:44:31.977204 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:31.977184    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:31.999055 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:31.999037    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 04:44:31.999228 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:31.999211    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 04:44:31.999312 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:31.999254    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 30 04:44:31.999410 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:31.999312    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 04:44:31.999410 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:31.999334    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 04:44:32.001471 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.000146    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 04:44:32.001471 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.000208    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 30 04:44:32.001471 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.000312    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 30 04:44:32.001471 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.000208    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 04:44:32.001471 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.000613    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 04:44:32.001471 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.000742    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 04:44:32.001798 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.001771    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 30 04:44:32.001846 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.001816    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 04:44:32.002477 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.002453    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 30 04:44:32.002564 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.002530    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 04:44:32.002617 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.002584    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 04:44:32.002829 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.002811    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 04:44:32.002915 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.002835    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 04:44:32.002985 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.002815    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bngnh\""
Oct 30 04:44:32.003045 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.002991    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 04:44:32.003095 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.003051    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 30 04:44:32.003142 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.003100    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 04:44:32.003190 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.003159    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 30 04:44:32.003237 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.003186    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-rdtp7\""
Oct 30 04:44:32.003387 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.003368    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 04:44:32.003449 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.003401    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 30 04:44:32.003495 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.003466    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 04:44:32.003548 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.003496    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 04:44:32.003548 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.003533    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 04:44:32.003636 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.003556    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 30 04:44:32.003636 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.003379    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-j5jgn\""
Oct 30 04:44:32.003745 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.003729    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 04:44:32.003797 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.003748    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 04:44:32.003797 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.003776    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 04:44:32.003881 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.003798    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 04:44:32.003941 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.003892    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 04:44:32.003941 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.003924    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 04:44:32.004025 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.003961    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-7hpknhdr5staj\""
Oct 30 04:44:32.004025 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.003981    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 30 04:44:32.004124 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.004025    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 30 04:44:32.004124 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.004070    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 04:44:32.004210 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.004162    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 04:44:32.004525 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.004510    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dzdcr\""
Oct 30 04:44:32.004708 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.004688    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fd380d55-276e-45eb-8191-c5940ed8385b-secret-thanos-querier-tls\") pod \"thanos-querier-95b457c5b-n92bg\" (UID: \"fd380d55-276e-45eb-8191-c5940ed8385b\") " pod="openshift-monitoring/thanos-querier-95b457c5b-n92bg"
Oct 30 04:44:32.004758 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.004726    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-hostroot\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:44:32.004803 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.004753    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-console-config\") pod \"console-7db77977b9-2f8qv\" (UID: \"47f03fac-07ba-4d50-9336-c5af8b4fbcc3\") " pod="openshift-console/console-7db77977b9-2f8qv"
Oct 30 04:44:32.004880 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.004864    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv4sg\" (UniqueName: \"kubernetes.io/projected/d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d-kube-api-access-gv4sg\") pod \"insights-runtime-extractor-srsjc\" (UID: \"d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d\") " pod="openshift-insights/insights-runtime-extractor-srsjc"
Oct 30 04:44:32.004951 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.004891    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ddad100f-6a08-4da8-a51a-f3f2fd5438bb-config-volume\") pod \"dns-default-bbk54\" (UID: \"ddad100f-6a08-4da8-a51a-f3f2fd5438bb\") " pod="openshift-dns/dns-default-bbk54"
Oct 30 04:44:32.004951 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.004930    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/76081e78-ba20-4f7c-b825-a9e0da23c511-node-exporter-textfile\") pod \"node-exporter-9mr8m\" (UID: \"76081e78-ba20-4f7c-b825-a9e0da23c511\") " pod="openshift-monitoring/node-exporter-9mr8m"
Oct 30 04:44:32.005065 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.004953    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/76081e78-ba20-4f7c-b825-a9e0da23c511-node-exporter-wtmp\") pod \"node-exporter-9mr8m\" (UID: \"76081e78-ba20-4f7c-b825-a9e0da23c511\") " pod="openshift-monitoring/node-exporter-9mr8m"
Oct 30 04:44:32.005065 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.004978    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qld8j\" (UniqueName: \"kubernetes.io/projected/7ad09519-e98b-4efc-8b6b-ec7d031c3c2f-kube-api-access-qld8j\") pod \"ingress-canary-jghzj\" (UID: \"7ad09519-e98b-4efc-8b6b-ec7d031c3c2f\") " pod="openshift-ingress-canary/ingress-canary-jghzj"
Oct 30 04:44:32.005065 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.005003    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pknnw\" (UniqueName: \"kubernetes.io/projected/70169887-cc72-4fb5-96b5-e09e260d7a8b-kube-api-access-pknnw\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:44:32.005065 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.005004    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-console-config\") pod \"console-7db77977b9-2f8qv\" (UID: \"47f03fac-07ba-4d50-9336-c5af8b4fbcc3\") " pod="openshift-console/console-7db77977b9-2f8qv"
Oct 30 04:44:32.005065 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.005032    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/76081e78-ba20-4f7c-b825-a9e0da23c511-node-exporter-textfile\") pod \"node-exporter-9mr8m\" (UID: \"76081e78-ba20-4f7c-b825-a9e0da23c511\") " pod="openshift-monitoring/node-exporter-9mr8m"
Oct 30 04:44:32.005065 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.005051    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-host-run-k8s-cni-cncf-io\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:44:32.005347 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.005052    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fd380d55-276e-45eb-8191-c5940ed8385b-secret-thanos-querier-tls\") pod \"thanos-querier-95b457c5b-n92bg\" (UID: \"fd380d55-276e-45eb-8191-c5940ed8385b\") " pod="openshift-monitoring/thanos-querier-95b457c5b-n92bg"
Oct 30 04:44:32.005347 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.005096    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/faa6a40d-2e9e-4348-a599-0df6f2d0177d-console-serving-cert\") pod \"console-f66c46c7c-skmxs\" (UID: \"faa6a40d-2e9e-4348-a599-0df6f2d0177d\") " pod="openshift-console/console-f66c46c7c-skmxs"
Oct 30 04:44:32.005347 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.005134    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ddad100f-6a08-4da8-a51a-f3f2fd5438bb-config-volume\") pod \"dns-default-bbk54\" (UID: \"ddad100f-6a08-4da8-a51a-f3f2fd5438bb\") " pod="openshift-dns/dns-default-bbk54"
Oct 30 04:44:32.005347 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.005172    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-etc-sysctl-d\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:44:32.005347 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.005210    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-host-slash\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:44:32.005347 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.005251    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-etc-openvswitch\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:44:32.005347 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.005290    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/70169887-cc72-4fb5-96b5-e09e260d7a8b-ovnkube-script-lib\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:44:32.005542 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.005355    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-service-ca\") pod \"console-7db77977b9-2f8qv\" (UID: \"47f03fac-07ba-4d50-9336-c5af8b4fbcc3\") " pod="openshift-console/console-7db77977b9-2f8qv"
Oct 30 04:44:32.005542 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.005428    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-etc-tuned\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:44:32.005542 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.005457    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-host-run-netns\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:44:32.005542 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.005487    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-multus-socket-dir-parent\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:44:32.005542 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.005527    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/41a4ec4b-b407-4316-9238-dee6be432d3c-dbus\") pod \"global-pull-secret-syncer-tt5mj\" (UID: \"41a4ec4b-b407-4316-9238-dee6be432d3c\") " pod="kube-system/global-pull-secret-syncer-tt5mj"
Oct 30 04:44:32.005719 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.005545    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-service-ca\") pod \"console-7db77977b9-2f8qv\" (UID: \"47f03fac-07ba-4d50-9336-c5af8b4fbcc3\") " pod="openshift-console/console-7db77977b9-2f8qv"
Oct 30 04:44:32.005719 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.005558    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2a3eb4b-a128-4649-bf77-78df2705f147-cnibin\") pod \"multus-additional-cni-plugins-tnwc6\" (UID: \"a2a3eb4b-a128-4649-bf77-78df2705f147\") " pod="openshift-multus/multus-additional-cni-plugins-tnwc6"
Oct 30 04:44:32.005719 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.005577    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-etc-tuned\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:44:32.005719 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.005597    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-trusted-ca-bundle\") pod \"console-7db77977b9-2f8qv\" (UID: \"47f03fac-07ba-4d50-9336-c5af8b4fbcc3\") " pod="openshift-console/console-7db77977b9-2f8qv"
Oct 30 04:44:32.005719 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.005630    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd380d55-276e-45eb-8191-c5940ed8385b-metrics-client-ca\") pod \"thanos-querier-95b457c5b-n92bg\" (UID: \"fd380d55-276e-45eb-8191-c5940ed8385b\") " pod="openshift-monitoring/thanos-querier-95b457c5b-n92bg"
Oct 30 04:44:32.005719 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.005715    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/76081e78-ba20-4f7c-b825-a9e0da23c511-node-exporter-tls\") pod \"node-exporter-9mr8m\" (UID: \"76081e78-ba20-4f7c-b825-a9e0da23c511\") " pod="openshift-monitoring/node-exporter-9mr8m"
Oct 30 04:44:32.006019 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.005733    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a2a3eb4b-a128-4649-bf77-78df2705f147-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tnwc6\" (UID: \"a2a3eb4b-a128-4649-bf77-78df2705f147\") " pod="openshift-multus/multus-additional-cni-plugins-tnwc6"
Oct 30 04:44:32.006019 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.005755    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwcw4\" (UniqueName: \"kubernetes.io/projected/885d476c-7568-4b63-ab9d-1da7608706f5-kube-api-access-kwcw4\") pod \"node-resolver-gndxs\" (UID: \"885d476c-7568-4b63-ab9d-1da7608706f5\") " pod="openshift-dns/node-resolver-gndxs"
Oct 30 04:44:32.006019 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.005769    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ad09519-e98b-4efc-8b6b-ec7d031c3c2f-cert\") pod \"ingress-canary-jghzj\" (UID: \"7ad09519-e98b-4efc-8b6b-ec7d031c3c2f\") " pod="openshift-ingress-canary/ingress-canary-jghzj"
Oct 30 04:44:32.006019 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.005784    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-log-socket\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:44:32.006019 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.005807    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29k58\" (UniqueName: \"kubernetes.io/projected/daf6535c-7f33-407a-a5d9-e1c9525c8cb8-kube-api-access-29k58\") pod \"node-ca-zqs6v\" (UID: \"daf6535c-7f33-407a-a5d9-e1c9525c8cb8\") " pod="openshift-image-registry/node-ca-zqs6v"
Oct 30 04:44:32.006019 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.005989    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a2a3eb4b-a128-4649-bf77-78df2705f147-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tnwc6\" (UID: \"a2a3eb4b-a128-4649-bf77-78df2705f147\") " pod="openshift-multus/multus-additional-cni-plugins-tnwc6"
Oct 30 04:44:32.006019 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.006015    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ad09519-e98b-4efc-8b6b-ec7d031c3c2f-cert\") pod \"ingress-canary-jghzj\" (UID: \"7ad09519-e98b-4efc-8b6b-ec7d031c3c2f\") " pod="openshift-ingress-canary/ingress-canary-jghzj"
Oct 30 04:44:32.006318 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.006047    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/76081e78-ba20-4f7c-b825-a9e0da23c511-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9mr8m\" (UID: \"76081e78-ba20-4f7c-b825-a9e0da23c511\") " pod="openshift-monitoring/node-exporter-9mr8m"
Oct 30 04:44:32.006318 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.006078    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eddb252a-1213-4eb9-9219-5a0abedf5587-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-r5tdl\" (UID: \"eddb252a-1213-4eb9-9219-5a0abedf5587\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-r5tdl"
Oct 30 04:44:32.006318 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.006115    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-lib-modules\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:44:32.006318 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.006142    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-host\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:44:32.006318 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.006167    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/672524df-2f52-455b-9423-a58f53c31129-etc-selinux\") pod \"aws-ebs-csi-driver-node-vhsc7\" (UID: \"672524df-2f52-455b-9423-a58f53c31129\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhsc7"
Oct 30 04:44:32.006318 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.006198    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-run-systemd\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:44:32.006318 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.006222    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fd380d55-276e-45eb-8191-c5940ed8385b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-95b457c5b-n92bg\" (UID: \"fd380d55-276e-45eb-8191-c5940ed8385b\") " pod="openshift-monitoring/thanos-querier-95b457c5b-n92bg"
Oct 30 04:44:32.006318 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.006230    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/76081e78-ba20-4f7c-b825-a9e0da23c511-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9mr8m\" (UID: \"76081e78-ba20-4f7c-b825-a9e0da23c511\") " pod="openshift-monitoring/node-exporter-9mr8m"
Oct 30 04:44:32.006318 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.006240    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/885d476c-7568-4b63-ab9d-1da7608706f5-tmp-dir\") pod \"node-resolver-gndxs\" (UID: \"885d476c-7568-4b63-ab9d-1da7608706f5\") " pod="openshift-dns/node-resolver-gndxs"
Oct 30 04:44:32.006318 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.006293    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/672524df-2f52-455b-9423-a58f53c31129-registration-dir\") pod \"aws-ebs-csi-driver-node-vhsc7\" (UID: \"672524df-2f52-455b-9423-a58f53c31129\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhsc7"
Oct 30 04:44:32.006318 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.006314    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/885d476c-7568-4b63-ab9d-1da7608706f5-tmp-dir\") pod \"node-resolver-gndxs\" (UID: \"885d476c-7568-4b63-ab9d-1da7608706f5\") " pod="openshift-dns/node-resolver-gndxs"
Oct 30 04:44:32.006672 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.006412    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-host-cni-netd\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:44:32.006672 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.006428    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fd380d55-276e-45eb-8191-c5940ed8385b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-95b457c5b-n92bg\" (UID: \"fd380d55-276e-45eb-8191-c5940ed8385b\") " pod="openshift-monitoring/thanos-querier-95b457c5b-n92bg"
Oct 30 04:44:32.006672 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.006446    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-host-var-lib-cni-multus\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:44:32.006672 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.006476    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/73a78a0d-d19e-4a96-b8ce-253d631da703-konnectivity-ca\") pod \"konnectivity-agent-6bpl4\" (UID: \"73a78a0d-d19e-4a96-b8ce-253d631da703\") " pod="kube-system/konnectivity-agent-6bpl4"
Oct 30 04:44:32.006672 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.006494    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l294n\" (UniqueName: \"kubernetes.io/projected/bff48d36-3909-4838-b499-66e5081bcc90-kube-api-access-l294n\") pod \"iptables-alerter-st55p\" (UID: \"bff48d36-3909-4838-b499-66e5081bcc90\") " pod="openshift-network-operator/iptables-alerter-st55p"
Oct 30 04:44:32.006672 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.006566    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a2a3eb4b-a128-4649-bf77-78df2705f147-cni-binary-copy\") pod \"multus-additional-cni-plugins-tnwc6\" (UID: \"a2a3eb4b-a128-4649-bf77-78df2705f147\") " pod="openshift-multus/multus-additional-cni-plugins-tnwc6"
Oct 30 04:44:32.006672 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.006587    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a2a3eb4b-a128-4649-bf77-78df2705f147-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tnwc6\" (UID: \"a2a3eb4b-a128-4649-bf77-78df2705f147\") " pod="openshift-multus/multus-additional-cni-plugins-tnwc6"
Oct 30 04:44:32.006672 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.006607    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/73a78a0d-d19e-4a96-b8ce-253d631da703-konnectivity-ca\") pod \"konnectivity-agent-6bpl4\" (UID: \"73a78a0d-d19e-4a96-b8ce-253d631da703\") " pod="kube-system/konnectivity-agent-6bpl4"
Oct 30 04:44:32.006944 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.006679    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76081e78-ba20-4f7c-b825-a9e0da23c511-metrics-client-ca\") pod \"node-exporter-9mr8m\" (UID: \"76081e78-ba20-4f7c-b825-a9e0da23c511\") " pod="openshift-monitoring/node-exporter-9mr8m"
Oct 30 04:44:32.006944 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.006706    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-multus-cni-dir\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:44:32.006944 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.006794    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bf49335f-7b8e-4212-8777-1575cd0929f1-cni-binary-copy\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:44:32.006944 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.006802    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a2a3eb4b-a128-4649-bf77-78df2705f147-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tnwc6\" (UID: \"a2a3eb4b-a128-4649-bf77-78df2705f147\") " pod="openshift-multus/multus-additional-cni-plugins-tnwc6"
Oct 30 04:44:32.006944 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.006822    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a2a3eb4b-a128-4649-bf77-78df2705f147-cni-binary-copy\") pod \"multus-additional-cni-plugins-tnwc6\" (UID: \"a2a3eb4b-a128-4649-bf77-78df2705f147\") " pod="openshift-multus/multus-additional-cni-plugins-tnwc6"
Oct 30 04:44:32.006944 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.006831    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-host-run-netns\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:44:32.006944 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.006861    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klznl\" (UniqueName: \"kubernetes.io/projected/eddb252a-1213-4eb9-9219-5a0abedf5587-kube-api-access-klznl\") pod \"monitoring-plugin-77f4f76594-r5tdl\" (UID: \"eddb252a-1213-4eb9-9219-5a0abedf5587\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-r5tdl"
Oct 30 04:44:32.006944 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.006918    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-etc-systemd\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:44:32.006944 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.006943    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bf49335f-7b8e-4212-8777-1575cd0929f1-cni-binary-copy\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:44:32.007212 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.007024    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fd380d55-276e-45eb-8191-c5940ed8385b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-95b457c5b-n92bg\" (UID: \"fd380d55-276e-45eb-8191-c5940ed8385b\") " pod="openshift-monitoring/thanos-querier-95b457c5b-n92bg"
Oct 30 04:44:32.007212 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.007073    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmbs5\" (UniqueName: \"kubernetes.io/projected/ddad100f-6a08-4da8-a51a-f3f2fd5438bb-kube-api-access-vmbs5\") pod \"dns-default-bbk54\" (UID: \"ddad100f-6a08-4da8-a51a-f3f2fd5438bb\") " pod="openshift-dns/dns-default-bbk54"
Oct 30 04:44:32.007212 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.007123    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-console-serving-cert\") pod \"console-7db77977b9-2f8qv\" (UID: \"47f03fac-07ba-4d50-9336-c5af8b4fbcc3\") " pod="openshift-console/console-7db77977b9-2f8qv"
Oct 30 04:44:32.007212 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.007193    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fd380d55-276e-45eb-8191-c5940ed8385b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-95b457c5b-n92bg\" (UID: \"fd380d55-276e-45eb-8191-c5940ed8385b\") " pod="openshift-monitoring/thanos-querier-95b457c5b-n92bg"
Oct 30 04:44:32.007348 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.007230    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-host-kubelet\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:44:32.007348 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.007271    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-srsjc\" (UID: \"d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d\") " pod="openshift-insights/insights-runtime-extractor-srsjc"
Oct 30 04:44:32.007348 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.007290    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd380d55-276e-45eb-8191-c5940ed8385b-tmp\") pod \"thanos-querier-95b457c5b-n92bg\" (UID: \"fd380d55-276e-45eb-8191-c5940ed8385b\") " pod="openshift-monitoring/thanos-querier-95b457c5b-n92bg"
Oct 30 04:44:32.007435 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.007344    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd380d55-276e-45eb-8191-c5940ed8385b-tmp\") pod \"thanos-querier-95b457c5b-n92bg\" (UID: \"fd380d55-276e-45eb-8191-c5940ed8385b\") " pod="openshift-monitoring/thanos-querier-95b457c5b-n92bg"
Oct 30 04:44:32.007479 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.007447    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/faa6a40d-2e9e-4348-a599-0df6f2d0177d-console-config\") pod \"console-f66c46c7c-skmxs\" (UID: \"faa6a40d-2e9e-4348-a599-0df6f2d0177d\") " pod="openshift-console/console-f66c46c7c-skmxs"
Oct 30 04:44:32.007479 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.007467    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-var-lib-kubelet\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:44:32.007578 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.007481    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/daf6535c-7f33-407a-a5d9-e1c9525c8cb8-host\") pod \"node-ca-zqs6v\" (UID: \"daf6535c-7f33-407a-a5d9-e1c9525c8cb8\") " pod="openshift-image-registry/node-ca-zqs6v"
Oct 30 04:44:32.007578 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.007497    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8x7k\" (UniqueName: \"kubernetes.io/projected/3b1cae8a-12cf-4b59-85f0-a63f324fc8e1-kube-api-access-f8x7k\") pod \"network-check-target-dgsdt\" (UID: \"3b1cae8a-12cf-4b59-85f0-a63f324fc8e1\") " pod="openshift-network-diagnostics/network-check-target-dgsdt"
Oct 30 04:44:32.007578 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.007513    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3023960e-7dab-48fe-926b-b016f2e2b5b7-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-4z6wz\" (UID: \"3023960e-7dab-48fe-926b-b016f2e2b5b7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4z6wz"
Oct 30 04:44:32.007578 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.007450    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-srsjc\" (UID: \"d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d\") " pod="openshift-insights/insights-runtime-extractor-srsjc"
Oct 30 04:44:32.007791 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.007607    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3023960e-7dab-48fe-926b-b016f2e2b5b7-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-4z6wz\" (UID: \"3023960e-7dab-48fe-926b-b016f2e2b5b7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4z6wz"
Oct 30 04:44:32.007791 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.007628    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/faa6a40d-2e9e-4348-a599-0df6f2d0177d-tmp\") pod \"console-f66c46c7c-skmxs\" (UID: \"faa6a40d-2e9e-4348-a599-0df6f2d0177d\") " pod="openshift-console/console-f66c46c7c-skmxs"
Oct 30 04:44:32.007791 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.007643    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/daf6535c-7f33-407a-a5d9-e1c9525c8cb8-serviceca\") pod \"node-ca-zqs6v\" (UID: \"daf6535c-7f33-407a-a5d9-e1c9525c8cb8\") " pod="openshift-image-registry/node-ca-zqs6v"
Oct 30 04:44:32.007791 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.007658    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-oauth-serving-cert\") pod \"console-7db77977b9-2f8qv\" (UID: \"47f03fac-07ba-4d50-9336-c5af8b4fbcc3\") " pod="openshift-console/console-7db77977b9-2f8qv"
Oct 30 04:44:32.007791 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.007665    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/faa6a40d-2e9e-4348-a599-0df6f2d0177d-console-config\") pod \"console-f66c46c7c-skmxs\" (UID: \"faa6a40d-2e9e-4348-a599-0df6f2d0177d\") " pod="openshift-console/console-f66c46c7c-skmxs"
Oct 30 04:44:32.007791 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.007674    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6rb2\" (UniqueName: \"kubernetes.io/projected/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-kube-api-access-z6rb2\") pod \"console-7db77977b9-2f8qv\" (UID: \"47f03fac-07ba-4d50-9336-c5af8b4fbcc3\") " pod="openshift-console/console-7db77977b9-2f8qv"
Oct 30 04:44:32.007791 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.007698    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49nsv\" (UniqueName: \"kubernetes.io/projected/7f75d879-5e31-41fd-838c-b1f0408cb45a-kube-api-access-49nsv\") pod \"network-metrics-daemon-vjfrn\" (UID: \"7f75d879-5e31-41fd-838c-b1f0408cb45a\") " pod="openshift-multus/network-metrics-daemon-vjfrn"
Oct 30 04:44:32.007791 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.007730    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/faa6a40d-2e9e-4348-a599-0df6f2d0177d-tmp\") pod \"console-f66c46c7c-skmxs\" (UID: \"faa6a40d-2e9e-4348-a599-0df6f2d0177d\") " pod="openshift-console/console-f66c46c7c-skmxs"
Oct 30 04:44:32.007791 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.007743    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-tmp\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:44:32.007791 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.007758    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3023960e-7dab-48fe-926b-b016f2e2b5b7-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-4z6wz\" (UID: \"3023960e-7dab-48fe-926b-b016f2e2b5b7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4z6wz"
Oct 30 04:44:32.007791 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.007768    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-srsjc\" (UID: \"d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d\") " pod="openshift-insights/insights-runtime-extractor-srsjc"
Oct 30 04:44:32.008260 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.007806    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bf49335f-7b8e-4212-8777-1575cd0929f1-multus-daemon-config\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:44:32.008260 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.007831    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-tmp\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:44:32.008260 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.007838    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3023960e-7dab-48fe-926b-b016f2e2b5b7-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-4z6wz\" (UID: \"3023960e-7dab-48fe-926b-b016f2e2b5b7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4z6wz"
Oct 30 04:44:32.008260 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.007832    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ddad100f-6a08-4da8-a51a-f3f2fd5438bb-metrics-tls\") pod \"dns-default-bbk54\" (UID: \"ddad100f-6a08-4da8-a51a-f3f2fd5438bb\") " pod="openshift-dns/dns-default-bbk54"
Oct 30 04:44:32.008260 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.007851    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-oauth-serving-cert\") pod \"console-7db77977b9-2f8qv\" (UID: \"47f03fac-07ba-4d50-9336-c5af8b4fbcc3\") " pod="openshift-console/console-7db77977b9-2f8qv"
Oct 30 04:44:32.008260 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.007881    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-console-oauth-config\") pod \"console-7db77977b9-2f8qv\" (UID: \"47f03fac-07ba-4d50-9336-c5af8b4fbcc3\") " pod="openshift-console/console-7db77977b9-2f8qv"
Oct 30 04:44:32.008260 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.007927    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/672524df-2f52-455b-9423-a58f53c31129-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vhsc7\" (UID: \"672524df-2f52-455b-9423-a58f53c31129\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhsc7"
Oct 30 04:44:32.008260 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.007955    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfwq8\" (UniqueName: \"kubernetes.io/projected/672524df-2f52-455b-9423-a58f53c31129-kube-api-access-bfwq8\") pod \"aws-ebs-csi-driver-node-vhsc7\" (UID: \"672524df-2f52-455b-9423-a58f53c31129\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhsc7"
Oct 30 04:44:32.008260 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.007983    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-system-cni-dir\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:44:32.008260 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.008022    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bf49335f-7b8e-4212-8777-1575cd0929f1-multus-daemon-config\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:44:32.008260 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.008029    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-console-oauth-config\") pod \"console-7db77977b9-2f8qv\" (UID: \"47f03fac-07ba-4d50-9336-c5af8b4fbcc3\") " pod="openshift-console/console-7db77977b9-2f8qv"
Oct 30 04:44:32.008260 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.008117    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhgw5\" (UniqueName: \"kubernetes.io/projected/3023960e-7dab-48fe-926b-b016f2e2b5b7-kube-api-access-mhgw5\") pod \"prometheus-operator-9cd89b74c-4z6wz\" (UID: \"3023960e-7dab-48fe-926b-b016f2e2b5b7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4z6wz"
Oct 30 04:44:32.008260 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.008225    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-etc-kubernetes\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:44:32.008260 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.008258    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/672524df-2f52-455b-9423-a58f53c31129-sys-fs\") pod \"aws-ebs-csi-driver-node-vhsc7\" (UID: \"672524df-2f52-455b-9423-a58f53c31129\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhsc7"
Oct 30 04:44:32.008631 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.008274    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/daf6535c-7f33-407a-a5d9-e1c9525c8cb8-serviceca\") pod \"node-ca-zqs6v\" (UID: \"daf6535c-7f33-407a-a5d9-e1c9525c8cb8\") " pod="openshift-image-registry/node-ca-zqs6v"
Oct 30 04:44:32.008631 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.008282    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-systemd-units\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:44:32.008631 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.008321    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d-data-volume\") pod \"insights-runtime-extractor-srsjc\" (UID: \"d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d\") " pod="openshift-insights/insights-runtime-extractor-srsjc"
Oct 30 04:44:32.008631 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.008346    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-host-var-lib-cni-bin\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:44:32.008631 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.008362    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-host-run-multus-certs\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:44:32.008631 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.008379    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/76081e78-ba20-4f7c-b825-a9e0da23c511-node-exporter-accelerators-collector-config\") pod \"node-exporter-9mr8m\" (UID: \"76081e78-ba20-4f7c-b825-a9e0da23c511\") " pod="openshift-monitoring/node-exporter-9mr8m"
Oct 30 04:44:32.008631 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.008417    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3023960e-7dab-48fe-926b-b016f2e2b5b7-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-4z6wz\" (UID: \"3023960e-7dab-48fe-926b-b016f2e2b5b7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4z6wz"
Oct 30 04:44:32.008631 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.008424    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d-data-volume\") pod \"insights-runtime-extractor-srsjc\" (UID: \"d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d\") " pod="openshift-insights/insights-runtime-extractor-srsjc"
Oct 30 04:44:32.008631 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.008452    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-host-run-ovn-kubernetes\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:44:32.008631 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.008477    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/70169887-cc72-4fb5-96b5-e09e260d7a8b-ovnkube-config\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:44:32.008631 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.008537    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/76081e78-ba20-4f7c-b825-a9e0da23c511-node-exporter-accelerators-collector-config\") pod \"node-exporter-9mr8m\" (UID: \"76081e78-ba20-4f7c-b825-a9e0da23c511\") " pod="openshift-monitoring/node-exporter-9mr8m"
Oct 30 04:44:32.008631 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.008545    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/76081e78-ba20-4f7c-b825-a9e0da23c511-root\") pod \"node-exporter-9mr8m\" (UID: \"76081e78-ba20-4f7c-b825-a9e0da23c511\") " pod="openshift-monitoring/node-exporter-9mr8m"
Oct 30 04:44:32.008631 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.008579    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/885d476c-7568-4b63-ab9d-1da7608706f5-hosts-file\") pod \"node-resolver-gndxs\" (UID: \"885d476c-7568-4b63-ab9d-1da7608706f5\") " pod="openshift-dns/node-resolver-gndxs"
Oct 30 04:44:32.008631 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.008602    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-run-ovn\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:44:32.008631 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.008612    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/70169887-cc72-4fb5-96b5-e09e260d7a8b-ovnkube-config\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:44:32.008631 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.008628    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-host-cni-bin\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:44:32.009135 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.008648    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-etc-kubernetes\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:44:32.009135 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.008670    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-etc-modprobe-d\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:44:32.009135 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.008684    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-etc-sysconfig\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:44:32.009135 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.008707    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/672524df-2f52-455b-9423-a58f53c31129-socket-dir\") pod \"aws-ebs-csi-driver-node-vhsc7\" (UID: \"672524df-2f52-455b-9423-a58f53c31129\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhsc7"
Oct 30 04:44:32.009135 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.008723    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/672524df-2f52-455b-9423-a58f53c31129-device-dir\") pod \"aws-ebs-csi-driver-node-vhsc7\" (UID: \"672524df-2f52-455b-9423-a58f53c31129\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhsc7"
Oct 30 04:44:32.009135 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.008752    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fd380d55-276e-45eb-8191-c5940ed8385b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-95b457c5b-n92bg\" (UID: \"fd380d55-276e-45eb-8191-c5940ed8385b\") " pod="openshift-monitoring/thanos-querier-95b457c5b-n92bg"
Oct 30 04:44:32.009135 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.008770    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-tmp\") pod \"console-7db77977b9-2f8qv\" (UID: \"47f03fac-07ba-4d50-9336-c5af8b4fbcc3\") " pod="openshift-console/console-7db77977b9-2f8qv"
Oct 30 04:44:32.009135 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.008807    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f75d879-5e31-41fd-838c-b1f0408cb45a-metrics-certs\") pod \"network-metrics-daemon-vjfrn\" (UID: \"7f75d879-5e31-41fd-838c-b1f0408cb45a\") " pod="openshift-multus/network-metrics-daemon-vjfrn"
Oct 30 04:44:32.009135 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.008831    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-tmp\") pod \"console-7db77977b9-2f8qv\" (UID: \"47f03fac-07ba-4d50-9336-c5af8b4fbcc3\") " pod="openshift-console/console-7db77977b9-2f8qv"
Oct 30 04:44:32.009135 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.008877    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fd380d55-276e-45eb-8191-c5940ed8385b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-95b457c5b-n92bg\" (UID: \"fd380d55-276e-45eb-8191-c5940ed8385b\") " pod="openshift-monitoring/thanos-querier-95b457c5b-n92bg"
Oct 30 04:44:32.009135 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.008928    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-host-var-lib-kubelet\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:44:32.009135 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.008958    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fd380d55-276e-45eb-8191-c5940ed8385b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-95b457c5b-n92bg\" (UID: \"fd380d55-276e-45eb-8191-c5940ed8385b\") " pod="openshift-monitoring/thanos-querier-95b457c5b-n92bg"
Oct 30 04:44:32.009135 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.008993    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/41a4ec4b-b407-4316-9238-dee6be432d3c-kubelet-config\") pod \"global-pull-secret-syncer-tt5mj\" (UID: \"41a4ec4b-b407-4316-9238-dee6be432d3c\") " pod="kube-system/global-pull-secret-syncer-tt5mj"
Oct 30 04:44:32.009135 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.009029    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fd380d55-276e-45eb-8191-c5940ed8385b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-95b457c5b-n92bg\" (UID: \"fd380d55-276e-45eb-8191-c5940ed8385b\") " pod="openshift-monitoring/thanos-querier-95b457c5b-n92bg"
Oct 30 04:44:32.009135 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.009031    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-etc-sysctl-conf\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:44:32.009135 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.009075    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-run-openvswitch\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:44:32.009135 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.009103    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-node-log\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:44:32.009135 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.009121    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d-crio-socket\") pod \"insights-runtime-extractor-srsjc\" (UID: \"d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d\") " pod="openshift-insights/insights-runtime-extractor-srsjc"
Oct 30 04:44:32.009773 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.009153    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7tw7\" (UniqueName: \"kubernetes.io/projected/fd380d55-276e-45eb-8191-c5940ed8385b-kube-api-access-t7tw7\") pod \"thanos-querier-95b457c5b-n92bg\" (UID: \"fd380d55-276e-45eb-8191-c5940ed8385b\") " pod="openshift-monitoring/thanos-querier-95b457c5b-n92bg"
Oct 30 04:44:32.009773 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.009183    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/76081e78-ba20-4f7c-b825-a9e0da23c511-sys\") pod \"node-exporter-9mr8m\" (UID: \"76081e78-ba20-4f7c-b825-a9e0da23c511\") " pod="openshift-monitoring/node-exporter-9mr8m"
Oct 30 04:44:32.009773 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.009203    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/faa6a40d-2e9e-4348-a599-0df6f2d0177d-oauth-serving-cert\") pod \"console-f66c46c7c-skmxs\" (UID: \"faa6a40d-2e9e-4348-a599-0df6f2d0177d\") " pod="openshift-console/console-f66c46c7c-skmxs"
Oct 30 04:44:32.009773 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.009221    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bff48d36-3909-4838-b499-66e5081bcc90-iptables-alerter-script\") pod \"iptables-alerter-st55p\" (UID: \"bff48d36-3909-4838-b499-66e5081bcc90\") " pod="openshift-network-operator/iptables-alerter-st55p"
Oct 30 04:44:32.009773 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.009300    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-sys\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:44:32.009773 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.009384    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-var-lib-openvswitch\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:44:32.009773 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.009427    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/faa6a40d-2e9e-4348-a599-0df6f2d0177d-oauth-serving-cert\") pod \"console-f66c46c7c-skmxs\" (UID: \"faa6a40d-2e9e-4348-a599-0df6f2d0177d\") " pod="openshift-console/console-f66c46c7c-skmxs"
Oct 30 04:44:32.009773 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.009423    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-cnibin\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:44:32.009773 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.009456    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-multus-conf-dir\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:44:32.009773 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.009484    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dg2h\" (UniqueName: \"kubernetes.io/projected/76081e78-ba20-4f7c-b825-a9e0da23c511-kube-api-access-4dg2h\") pod \"node-exporter-9mr8m\" (UID: \"76081e78-ba20-4f7c-b825-a9e0da23c511\") " pod="openshift-monitoring/node-exporter-9mr8m"
Oct 30 04:44:32.009773 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.009511    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/41a4ec4b-b407-4316-9238-dee6be432d3c-original-pull-secret\") pod \"global-pull-secret-syncer-tt5mj\" (UID: \"41a4ec4b-b407-4316-9238-dee6be432d3c\") " pod="kube-system/global-pull-secret-syncer-tt5mj"
Oct 30 04:44:32.009773 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.009534    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxpck\" (UniqueName: \"kubernetes.io/projected/a2a3eb4b-a128-4649-bf77-78df2705f147-kube-api-access-nxpck\") pod \"multus-additional-cni-plugins-tnwc6\" (UID: \"a2a3eb4b-a128-4649-bf77-78df2705f147\") " pod="openshift-multus/multus-additional-cni-plugins-tnwc6"
Oct 30 04:44:32.009773 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.009550    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5q8q\" (UniqueName: \"kubernetes.io/projected/faa6a40d-2e9e-4348-a599-0df6f2d0177d-kube-api-access-p5q8q\") pod \"console-f66c46c7c-skmxs\" (UID: \"faa6a40d-2e9e-4348-a599-0df6f2d0177d\") " pod="openshift-console/console-f66c46c7c-skmxs"
Oct 30 04:44:32.009773 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.009576    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-run\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:44:32.009773 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.009608    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fd380d55-276e-45eb-8191-c5940ed8385b-secret-grpc-tls\") pod \"thanos-querier-95b457c5b-n92bg\" (UID: \"fd380d55-276e-45eb-8191-c5940ed8385b\") " pod="openshift-monitoring/thanos-querier-95b457c5b-n92bg"
Oct 30 04:44:32.009773 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.009624    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2a3eb4b-a128-4649-bf77-78df2705f147-system-cni-dir\") pod \"multus-additional-cni-plugins-tnwc6\" (UID: \"a2a3eb4b-a128-4649-bf77-78df2705f147\") " pod="openshift-multus/multus-additional-cni-plugins-tnwc6"
Oct 30 04:44:32.009773 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.009642    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/faa6a40d-2e9e-4348-a599-0df6f2d0177d-console-oauth-config\") pod \"console-f66c46c7c-skmxs\" (UID: \"faa6a40d-2e9e-4348-a599-0df6f2d0177d\") " pod="openshift-console/console-f66c46c7c-skmxs"
Oct 30 04:44:32.009773 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.009664    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/faa6a40d-2e9e-4348-a599-0df6f2d0177d-service-ca\") pod \"console-f66c46c7c-skmxs\" (UID: \"faa6a40d-2e9e-4348-a599-0df6f2d0177d\") " pod="openshift-console/console-f66c46c7c-skmxs"
Oct 30 04:44:32.009773 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.009726    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92qd4\" (UniqueName: \"kubernetes.io/projected/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-kube-api-access-92qd4\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:44:32.009773 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.009760    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/73a78a0d-d19e-4a96-b8ce-253d631da703-agent-certs\") pod \"konnectivity-agent-6bpl4\" (UID: \"73a78a0d-d19e-4a96-b8ce-253d631da703\") " pod="kube-system/konnectivity-agent-6bpl4"
Oct 30 04:44:32.009773 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.009761    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/41a4ec4b-b407-4316-9238-dee6be432d3c-original-pull-secret\") pod \"global-pull-secret-syncer-tt5mj\" (UID: \"41a4ec4b-b407-4316-9238-dee6be432d3c\") " pod="kube-system/global-pull-secret-syncer-tt5mj"
Oct 30 04:44:32.010466 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.009847    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/faa6a40d-2e9e-4348-a599-0df6f2d0177d-service-ca\") pod \"console-f66c46c7c-skmxs\" (UID: \"faa6a40d-2e9e-4348-a599-0df6f2d0177d\") " pod="openshift-console/console-f66c46c7c-skmxs"
Oct 30 04:44:32.010466 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.009852    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/70169887-cc72-4fb5-96b5-e09e260d7a8b-env-overrides\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:44:32.010466 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.009876    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/faa6a40d-2e9e-4348-a599-0df6f2d0177d-console-oauth-config\") pod \"console-f66c46c7c-skmxs\" (UID: \"faa6a40d-2e9e-4348-a599-0df6f2d0177d\") " pod="openshift-console/console-f66c46c7c-skmxs"
Oct 30 04:44:32.010466 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.009896    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w8n4\" (UniqueName: \"kubernetes.io/projected/bf49335f-7b8e-4212-8777-1575cd0929f1-kube-api-access-4w8n4\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:44:32.010466 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.009955    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddad100f-6a08-4da8-a51a-f3f2fd5438bb-tmp-dir\") pod \"dns-default-bbk54\" (UID: \"ddad100f-6a08-4da8-a51a-f3f2fd5438bb\") " pod="openshift-dns/dns-default-bbk54"
Oct 30 04:44:32.010466 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.009971    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fd380d55-276e-45eb-8191-c5940ed8385b-secret-grpc-tls\") pod \"thanos-querier-95b457c5b-n92bg\" (UID: \"fd380d55-276e-45eb-8191-c5940ed8385b\") " pod="openshift-monitoring/thanos-querier-95b457c5b-n92bg"
Oct 30 04:44:32.010466 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.009980    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bff48d36-3909-4838-b499-66e5081bcc90-host-slash\") pod \"iptables-alerter-st55p\" (UID: \"bff48d36-3909-4838-b499-66e5081bcc90\") " pod="openshift-network-operator/iptables-alerter-st55p"
Oct 30 04:44:32.010466 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.009997    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/70169887-cc72-4fb5-96b5-e09e260d7a8b-env-overrides\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:44:32.010466 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.010005    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:44:32.010466 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.010049    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddad100f-6a08-4da8-a51a-f3f2fd5438bb-tmp-dir\") pod \"dns-default-bbk54\" (UID: \"ddad100f-6a08-4da8-a51a-f3f2fd5438bb\") " pod="openshift-dns/dns-default-bbk54"
Oct 30 04:44:32.010466 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.010059    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/70169887-cc72-4fb5-96b5-e09e260d7a8b-ovn-node-metrics-cert\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:44:32.010466 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.010095    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2a3eb4b-a128-4649-bf77-78df2705f147-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tnwc6\" (UID: \"a2a3eb4b-a128-4649-bf77-78df2705f147\") " pod="openshift-multus/multus-additional-cni-plugins-tnwc6"
Oct 30 04:44:32.010822 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.010580    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 30 04:44:32.011508 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.011320    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 04:44:32.011508 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.011339    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tg42c\""
Oct 30 04:44:32.011508 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.011350    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 04:44:32.011508 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.011361    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-k8wtk\""
Oct 30 04:44:32.011508 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.011342    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 04:44:32.011508 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.011384    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 04:44:32.011508 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.011365    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 04:44:32.011508 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.011367    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4jtzx\""
Oct 30 04:44:32.011508 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.011404    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-g782k\""
Oct 30 04:44:32.011508 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.011405    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-g88vf\""
Oct 30 04:44:32.011508 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.011376    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k257x\""
Oct 30 04:44:32.011508 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.011423    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 04:44:32.011508 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.011433    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 04:44:32.011508 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.011445    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rhd5l\""
Oct 30 04:44:32.011508 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.011475    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-q24mq\""
Oct 30 04:44:32.011508 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.011483    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 04:44:32.011508 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.011474    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-fmr9k\""
Oct 30 04:44:32.012149 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.011549    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 04:44:32.012149 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.011567    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-tdqqv\""
Oct 30 04:44:32.012149 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.011834    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 30 04:44:32.012149 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.011845    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 04:44:32.012149 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.011911    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-bxf29\""
Oct 30 04:44:32.012149 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.011930    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-jzfzk\""
Oct 30 04:44:32.012149 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.011974    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 04:44:32.012461 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.012447    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 30 04:44:32.015554 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.015539    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/faa6a40d-2e9e-4348-a599-0df6f2d0177d-console-serving-cert\") pod \"console-f66c46c7c-skmxs\" (UID: \"faa6a40d-2e9e-4348-a599-0df6f2d0177d\") " pod="openshift-console/console-f66c46c7c-skmxs"
Oct 30 04:44:32.015661 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.015646    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/70169887-cc72-4fb5-96b5-e09e260d7a8b-ovnkube-script-lib\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:44:32.015924 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.015891    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd380d55-276e-45eb-8191-c5940ed8385b-metrics-client-ca\") pod \"thanos-querier-95b457c5b-n92bg\" (UID: \"fd380d55-276e-45eb-8191-c5940ed8385b\") " pod="openshift-monitoring/thanos-querier-95b457c5b-n92bg"
Oct 30 04:44:32.015958 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.015937    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-trusted-ca-bundle\") pod \"console-7db77977b9-2f8qv\" (UID: \"47f03fac-07ba-4d50-9336-c5af8b4fbcc3\") " pod="openshift-console/console-7db77977b9-2f8qv"
Oct 30 04:44:32.015994 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.015960    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/76081e78-ba20-4f7c-b825-a9e0da23c511-node-exporter-tls\") pod \"node-exporter-9mr8m\" (UID: \"76081e78-ba20-4f7c-b825-a9e0da23c511\") " pod="openshift-monitoring/node-exporter-9mr8m"
Oct 30 04:44:32.016680 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.016665    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eddb252a-1213-4eb9-9219-5a0abedf5587-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-r5tdl\" (UID: \"eddb252a-1213-4eb9-9219-5a0abedf5587\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-r5tdl"
Oct 30 04:44:32.017796 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.017782    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-console-serving-cert\") pod \"console-7db77977b9-2f8qv\" (UID: \"47f03fac-07ba-4d50-9336-c5af8b4fbcc3\") " pod="openshift-console/console-7db77977b9-2f8qv"
Oct 30 04:44:32.017845 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.017806    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76081e78-ba20-4f7c-b825-a9e0da23c511-metrics-client-ca\") pod \"node-exporter-9mr8m\" (UID: \"76081e78-ba20-4f7c-b825-a9e0da23c511\") " pod="openshift-monitoring/node-exporter-9mr8m"
Oct 30 04:44:32.018986 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.018971    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3023960e-7dab-48fe-926b-b016f2e2b5b7-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-4z6wz\" (UID: \"3023960e-7dab-48fe-926b-b016f2e2b5b7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4z6wz"
Oct 30 04:44:32.019048 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.019036    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-srsjc\" (UID: \"d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d\") " pod="openshift-insights/insights-runtime-extractor-srsjc"
Oct 30 04:44:32.019080 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.019044    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ddad100f-6a08-4da8-a51a-f3f2fd5438bb-metrics-tls\") pod \"dns-default-bbk54\" (UID: \"ddad100f-6a08-4da8-a51a-f3f2fd5438bb\") " pod="openshift-dns/dns-default-bbk54"
Oct 30 04:44:32.019109 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.019100    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f75d879-5e31-41fd-838c-b1f0408cb45a-metrics-certs\") pod \"network-metrics-daemon-vjfrn\" (UID: \"7f75d879-5e31-41fd-838c-b1f0408cb45a\") " pod="openshift-multus/network-metrics-daemon-vjfrn"
Oct 30 04:44:32.020126 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.020112    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bff48d36-3909-4838-b499-66e5081bcc90-iptables-alerter-script\") pod \"iptables-alerter-st55p\" (UID: \"bff48d36-3909-4838-b499-66e5081bcc90\") " pod="openshift-network-operator/iptables-alerter-st55p"
Oct 30 04:44:32.020204 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.020193    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/73a78a0d-d19e-4a96-b8ce-253d631da703-agent-certs\") pod \"konnectivity-agent-6bpl4\" (UID: \"73a78a0d-d19e-4a96-b8ce-253d631da703\") " pod="kube-system/konnectivity-agent-6bpl4"
Oct 30 04:44:32.020334 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.020323    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/70169887-cc72-4fb5-96b5-e09e260d7a8b-ovn-node-metrics-cert\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:44:32.022836 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.022816    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv4sg\" (UniqueName: \"kubernetes.io/projected/d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d-kube-api-access-gv4sg\") pod \"insights-runtime-extractor-srsjc\" (UID: \"d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d\") " pod="openshift-insights/insights-runtime-extractor-srsjc"
Oct 30 04:44:32.023289 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.023273    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qld8j\" (UniqueName: \"kubernetes.io/projected/7ad09519-e98b-4efc-8b6b-ec7d031c3c2f-kube-api-access-qld8j\") pod \"ingress-canary-jghzj\" (UID: \"7ad09519-e98b-4efc-8b6b-ec7d031c3c2f\") " pod="openshift-ingress-canary/ingress-canary-jghzj"
Oct 30 04:44:32.023478 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.023464    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwcw4\" (UniqueName: \"kubernetes.io/projected/885d476c-7568-4b63-ab9d-1da7608706f5-kube-api-access-kwcw4\") pod \"node-resolver-gndxs\" (UID: \"885d476c-7568-4b63-ab9d-1da7608706f5\") " pod="openshift-dns/node-resolver-gndxs"
Oct 30 04:44:32.023674 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.023657    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pknnw\" (UniqueName: \"kubernetes.io/projected/70169887-cc72-4fb5-96b5-e09e260d7a8b-kube-api-access-pknnw\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:44:32.023958 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.023943    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l294n\" (UniqueName: \"kubernetes.io/projected/bff48d36-3909-4838-b499-66e5081bcc90-kube-api-access-l294n\") pod \"iptables-alerter-st55p\" (UID: \"bff48d36-3909-4838-b499-66e5081bcc90\") " pod="openshift-network-operator/iptables-alerter-st55p"
Oct 30 04:44:32.024182 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.024168    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29k58\" (UniqueName: \"kubernetes.io/projected/daf6535c-7f33-407a-a5d9-e1c9525c8cb8-kube-api-access-29k58\") pod \"node-ca-zqs6v\" (UID: \"daf6535c-7f33-407a-a5d9-e1c9525c8cb8\") " pod="openshift-image-registry/node-ca-zqs6v"
Oct 30 04:44:32.031413 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.031397    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dg2h\" (UniqueName: \"kubernetes.io/projected/76081e78-ba20-4f7c-b825-a9e0da23c511-kube-api-access-4dg2h\") pod \"node-exporter-9mr8m\" (UID: \"76081e78-ba20-4f7c-b825-a9e0da23c511\") " pod="openshift-monitoring/node-exporter-9mr8m"
Oct 30 04:44:32.031643 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.031626    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5q8q\" (UniqueName: \"kubernetes.io/projected/faa6a40d-2e9e-4348-a599-0df6f2d0177d-kube-api-access-p5q8q\") pod \"console-f66c46c7c-skmxs\" (UID: \"faa6a40d-2e9e-4348-a599-0df6f2d0177d\") " pod="openshift-console/console-f66c46c7c-skmxs"
Oct 30 04:44:32.032168 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.032154    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfwq8\" (UniqueName: \"kubernetes.io/projected/672524df-2f52-455b-9423-a58f53c31129-kube-api-access-bfwq8\") pod \"aws-ebs-csi-driver-node-vhsc7\" (UID: \"672524df-2f52-455b-9423-a58f53c31129\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhsc7"
Oct 30 04:44:32.032336 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.032321    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhgw5\" (UniqueName: \"kubernetes.io/projected/3023960e-7dab-48fe-926b-b016f2e2b5b7-kube-api-access-mhgw5\") pod \"prometheus-operator-9cd89b74c-4z6wz\" (UID: \"3023960e-7dab-48fe-926b-b016f2e2b5b7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4z6wz"
Oct 30 04:44:32.032403 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.032387    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmbs5\" (UniqueName: \"kubernetes.io/projected/ddad100f-6a08-4da8-a51a-f3f2fd5438bb-kube-api-access-vmbs5\") pod \"dns-default-bbk54\" (UID: \"ddad100f-6a08-4da8-a51a-f3f2fd5438bb\") " pod="openshift-dns/dns-default-bbk54"
Oct 30 04:44:32.033081 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.033066    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7tw7\" (UniqueName: \"kubernetes.io/projected/fd380d55-276e-45eb-8191-c5940ed8385b-kube-api-access-t7tw7\") pod \"thanos-querier-95b457c5b-n92bg\" (UID: \"fd380d55-276e-45eb-8191-c5940ed8385b\") " pod="openshift-monitoring/thanos-querier-95b457c5b-n92bg"
Oct 30 04:44:32.033125 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.033112    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klznl\" (UniqueName: \"kubernetes.io/projected/eddb252a-1213-4eb9-9219-5a0abedf5587-kube-api-access-klznl\") pod \"monitoring-plugin-77f4f76594-r5tdl\" (UID: \"eddb252a-1213-4eb9-9219-5a0abedf5587\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-r5tdl"
Oct 30 04:44:32.034024 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.034004    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8x7k\" (UniqueName: \"kubernetes.io/projected/3b1cae8a-12cf-4b59-85f0-a63f324fc8e1-kube-api-access-f8x7k\") pod \"network-check-target-dgsdt\" (UID: \"3b1cae8a-12cf-4b59-85f0-a63f324fc8e1\") " pod="openshift-network-diagnostics/network-check-target-dgsdt"
Oct 30 04:44:32.034081 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.034006    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6rb2\" (UniqueName: \"kubernetes.io/projected/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-kube-api-access-z6rb2\") pod \"console-7db77977b9-2f8qv\" (UID: \"47f03fac-07ba-4d50-9336-c5af8b4fbcc3\") " pod="openshift-console/console-7db77977b9-2f8qv"
Oct 30 04:44:32.034292 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.034277    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92qd4\" (UniqueName: \"kubernetes.io/projected/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-kube-api-access-92qd4\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:44:32.034364 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.034349    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w8n4\" (UniqueName: \"kubernetes.io/projected/bf49335f-7b8e-4212-8777-1575cd0929f1-kube-api-access-4w8n4\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:44:32.034707 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.034689    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49nsv\" (UniqueName: \"kubernetes.io/projected/7f75d879-5e31-41fd-838c-b1f0408cb45a-kube-api-access-49nsv\") pod \"network-metrics-daemon-vjfrn\" (UID: \"7f75d879-5e31-41fd-838c-b1f0408cb45a\") " pod="openshift-multus/network-metrics-daemon-vjfrn"
Oct 30 04:44:32.034743 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.034688    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxpck\" (UniqueName: \"kubernetes.io/projected/a2a3eb4b-a128-4649-bf77-78df2705f147-kube-api-access-nxpck\") pod \"multus-additional-cni-plugins-tnwc6\" (UID: \"a2a3eb4b-a128-4649-bf77-78df2705f147\") " pod="openshift-multus/multus-additional-cni-plugins-tnwc6"
Oct 30 04:44:32.077633 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.077610    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:32.111031 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.110955    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-multus-cni-dir\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:44:32.111031 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.110987    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-host-run-netns\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:44:32.111031 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111003    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-etc-systemd\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:44:32.111031 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111018    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-host-kubelet\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:44:32.111031 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111036    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-var-lib-kubelet\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:44:32.111403 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111052    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/daf6535c-7f33-407a-a5d9-e1c9525c8cb8-host\") pod \"node-ca-zqs6v\" (UID: \"daf6535c-7f33-407a-a5d9-e1c9525c8cb8\") " pod="openshift-image-registry/node-ca-zqs6v"
Oct 30 04:44:32.111403 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111071    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/672524df-2f52-455b-9423-a58f53c31129-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vhsc7\" (UID: \"672524df-2f52-455b-9423-a58f53c31129\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhsc7"
Oct 30 04:44:32.111403 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111080    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-host-run-netns\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:44:32.111403 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111097    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-host-kubelet\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:44:32.111403 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111106    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-etc-systemd\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:44:32.111403 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111092    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-system-cni-dir\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:44:32.111403 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111135    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/672524df-2f52-455b-9423-a58f53c31129-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vhsc7\" (UID: \"672524df-2f52-455b-9423-a58f53c31129\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhsc7"
Oct 30 04:44:32.111403 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111136    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-var-lib-kubelet\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:44:32.111403 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111150    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-system-cni-dir\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:44:32.111403 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111140    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/daf6535c-7f33-407a-a5d9-e1c9525c8cb8-host\") pod \"node-ca-zqs6v\" (UID: \"daf6535c-7f33-407a-a5d9-e1c9525c8cb8\") " pod="openshift-image-registry/node-ca-zqs6v"
Oct 30 04:44:32.111403 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111172    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-etc-kubernetes\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:44:32.111403 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111085    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-multus-cni-dir\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:44:32.111403 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111218    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-etc-kubernetes\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:44:32.111403 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111231    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/672524df-2f52-455b-9423-a58f53c31129-sys-fs\") pod \"aws-ebs-csi-driver-node-vhsc7\" (UID: \"672524df-2f52-455b-9423-a58f53c31129\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhsc7"
Oct 30 04:44:32.111403 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111270    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-systemd-units\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:44:32.111403 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111294    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/672524df-2f52-455b-9423-a58f53c31129-sys-fs\") pod \"aws-ebs-csi-driver-node-vhsc7\" (UID: \"672524df-2f52-455b-9423-a58f53c31129\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhsc7"
Oct 30 04:44:32.111403 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111332    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-host-var-lib-cni-bin\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:44:32.111403 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111361    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-host-run-multus-certs\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:44:32.111403 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111365    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-host-var-lib-cni-bin\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:44:32.111403 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111373    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-systemd-units\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:44:32.111403 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111405    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-host-run-multus-certs\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:44:32.112233 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111431    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-host-run-ovn-kubernetes\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:44:32.112233 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111451    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/76081e78-ba20-4f7c-b825-a9e0da23c511-root\") pod \"node-exporter-9mr8m\" (UID: \"76081e78-ba20-4f7c-b825-a9e0da23c511\") " pod="openshift-monitoring/node-exporter-9mr8m"
Oct 30 04:44:32.112233 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111453    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-host-run-ovn-kubernetes\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:44:32.112233 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111467    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/885d476c-7568-4b63-ab9d-1da7608706f5-hosts-file\") pod \"node-resolver-gndxs\" (UID: \"885d476c-7568-4b63-ab9d-1da7608706f5\") " pod="openshift-dns/node-resolver-gndxs"
Oct 30 04:44:32.112233 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111489    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/76081e78-ba20-4f7c-b825-a9e0da23c511-root\") pod \"node-exporter-9mr8m\" (UID: \"76081e78-ba20-4f7c-b825-a9e0da23c511\") " pod="openshift-monitoring/node-exporter-9mr8m"
Oct 30 04:44:32.112233 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111501    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-run-ovn\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:44:32.112233 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111527    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-host-cni-bin\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:44:32.112233 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111536    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-run-ovn\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:44:32.112233 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111537    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/885d476c-7568-4b63-ab9d-1da7608706f5-hosts-file\") pod \"node-resolver-gndxs\" (UID: \"885d476c-7568-4b63-ab9d-1da7608706f5\") " pod="openshift-dns/node-resolver-gndxs"
Oct 30 04:44:32.112233 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111544    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-etc-kubernetes\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:44:32.112233 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111570    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-etc-modprobe-d\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:44:32.112233 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111586    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-etc-sysconfig\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:44:32.112233 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111602    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/672524df-2f52-455b-9423-a58f53c31129-socket-dir\") pod \"aws-ebs-csi-driver-node-vhsc7\" (UID: \"672524df-2f52-455b-9423-a58f53c31129\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhsc7"
Oct 30 04:44:32.112233 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111619    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/672524df-2f52-455b-9423-a58f53c31129-device-dir\") pod \"aws-ebs-csi-driver-node-vhsc7\" (UID: \"672524df-2f52-455b-9423-a58f53c31129\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhsc7"
Oct 30 04:44:32.112233 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111569    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-etc-kubernetes\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:44:32.112233 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111645    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-host-var-lib-kubelet\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:44:32.112233 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111658    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-etc-sysconfig\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:44:32.112233 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111671    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/41a4ec4b-b407-4316-9238-dee6be432d3c-kubelet-config\") pod \"global-pull-secret-syncer-tt5mj\" (UID: \"41a4ec4b-b407-4316-9238-dee6be432d3c\") " pod="kube-system/global-pull-secret-syncer-tt5mj"
Oct 30 04:44:32.112233 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111674    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-host-var-lib-kubelet\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:44:32.112233 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111588    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-host-cni-bin\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:44:32.112233 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111709    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-etc-sysctl-conf\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:44:32.112233 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111709    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/672524df-2f52-455b-9423-a58f53c31129-device-dir\") pod \"aws-ebs-csi-driver-node-vhsc7\" (UID: \"672524df-2f52-455b-9423-a58f53c31129\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhsc7"
Oct 30 04:44:32.112233 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111713    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/672524df-2f52-455b-9423-a58f53c31129-socket-dir\") pod \"aws-ebs-csi-driver-node-vhsc7\" (UID: \"672524df-2f52-455b-9423-a58f53c31129\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhsc7"
Oct 30 04:44:32.112233 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111720    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/41a4ec4b-b407-4316-9238-dee6be432d3c-kubelet-config\") pod \"global-pull-secret-syncer-tt5mj\" (UID: \"41a4ec4b-b407-4316-9238-dee6be432d3c\") " pod="kube-system/global-pull-secret-syncer-tt5mj"
Oct 30 04:44:32.112233 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111730    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-run-openvswitch\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:44:32.112233 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111747    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-etc-modprobe-d\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:44:32.112233 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111760    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-run-openvswitch\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:44:32.112233 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111753    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-node-log\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:44:32.112233 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111777    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-node-log\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:44:32.112233 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111799    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-etc-sysctl-conf\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:44:32.112233 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111800    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d-crio-socket\") pod \"insights-runtime-extractor-srsjc\" (UID: \"d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d\") " pod="openshift-insights/insights-runtime-extractor-srsjc"
Oct 30 04:44:32.112233 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111828    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/76081e78-ba20-4f7c-b825-a9e0da23c511-sys\") pod \"node-exporter-9mr8m\" (UID: \"76081e78-ba20-4f7c-b825-a9e0da23c511\") " pod="openshift-monitoring/node-exporter-9mr8m"
Oct 30 04:44:32.112233 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111852    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-sys\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:44:32.112233 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111878    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/76081e78-ba20-4f7c-b825-a9e0da23c511-sys\") pod \"node-exporter-9mr8m\" (UID: \"76081e78-ba20-4f7c-b825-a9e0da23c511\") " pod="openshift-monitoring/node-exporter-9mr8m"
Oct 30 04:44:32.112233 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111882    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d-crio-socket\") pod \"insights-runtime-extractor-srsjc\" (UID: \"d7ae9535-d5d2-4bdc-8d4e-cb994a71f48d\") " pod="openshift-insights/insights-runtime-extractor-srsjc"
Oct 30 04:44:32.112233 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111876    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-var-lib-openvswitch\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:44:32.112233 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111921    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-sys\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:44:32.112233 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111921    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-cnibin\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:44:32.112233 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111950    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-multus-conf-dir\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:44:32.112233 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111966    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-var-lib-openvswitch\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:44:32.112233 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111979    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-multus-conf-dir\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:44:32.112233 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111971    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-run\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:44:32.112233 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111992    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-run\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:44:32.112233 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.111950    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-cnibin\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:44:32.112233 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.112014    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2a3eb4b-a128-4649-bf77-78df2705f147-system-cni-dir\") pod \"multus-additional-cni-plugins-tnwc6\" (UID: \"a2a3eb4b-a128-4649-bf77-78df2705f147\") " pod="openshift-multus/multus-additional-cni-plugins-tnwc6"
Oct 30 04:44:32.112233 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.112043    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bff48d36-3909-4838-b499-66e5081bcc90-host-slash\") pod \"iptables-alerter-st55p\" (UID: \"bff48d36-3909-4838-b499-66e5081bcc90\") " pod="openshift-network-operator/iptables-alerter-st55p"
Oct 30 04:44:32.112233 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.112071    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2a3eb4b-a128-4649-bf77-78df2705f147-system-cni-dir\") pod \"multus-additional-cni-plugins-tnwc6\" (UID: \"a2a3eb4b-a128-4649-bf77-78df2705f147\") " pod="openshift-multus/multus-additional-cni-plugins-tnwc6"
Oct 30 04:44:32.112233 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.112093    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:44:32.112233 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.112122    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2a3eb4b-a128-4649-bf77-78df2705f147-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tnwc6\" (UID: \"a2a3eb4b-a128-4649-bf77-78df2705f147\") " pod="openshift-multus/multus-additional-cni-plugins-tnwc6"
Oct 30 04:44:32.112233 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.112121    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bff48d36-3909-4838-b499-66e5081bcc90-host-slash\") pod \"iptables-alerter-st55p\" (UID: \"bff48d36-3909-4838-b499-66e5081bcc90\") " pod="openshift-network-operator/iptables-alerter-st55p"
Oct 30 04:44:32.112233 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.112148    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-hostroot\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:44:32.112233 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.112172    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2a3eb4b-a128-4649-bf77-78df2705f147-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tnwc6\" (UID: \"a2a3eb4b-a128-4649-bf77-78df2705f147\") " pod="openshift-multus/multus-additional-cni-plugins-tnwc6"
Oct 30 04:44:32.112233 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.112176    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/76081e78-ba20-4f7c-b825-a9e0da23c511-node-exporter-wtmp\") pod \"node-exporter-9mr8m\" (UID: \"76081e78-ba20-4f7c-b825-a9e0da23c511\") " pod="openshift-monitoring/node-exporter-9mr8m"
Oct 30 04:44:32.112233 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.112190    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:44:32.112233 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.112204    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-host-run-k8s-cni-cncf-io\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:44:32.112233 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.112217    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-hostroot\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:44:32.112233 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.112228    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-etc-sysctl-d\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:44:32.112233 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.112254    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-host-slash\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:44:32.112233 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.112257    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-host-run-k8s-cni-cncf-io\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:44:32.112233 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.112257    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/76081e78-ba20-4f7c-b825-a9e0da23c511-node-exporter-wtmp\") pod \"node-exporter-9mr8m\" (UID: \"76081e78-ba20-4f7c-b825-a9e0da23c511\") " pod="openshift-monitoring/node-exporter-9mr8m"
Oct 30 04:44:32.112233 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.112278    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-etc-openvswitch\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:44:32.114160 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.112302    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-host-slash\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:44:32.114160 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.112311    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-etc-openvswitch\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:44:32.114160 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.112303    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-host-run-netns\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:44:32.114160 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.112337    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-host-run-netns\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:44:32.114160 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.112322    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-etc-sysctl-d\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:44:32.114160 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.112347    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-multus-socket-dir-parent\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:44:32.114160 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.112377    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/41a4ec4b-b407-4316-9238-dee6be432d3c-dbus\") pod \"global-pull-secret-syncer-tt5mj\" (UID: \"41a4ec4b-b407-4316-9238-dee6be432d3c\") " pod="kube-system/global-pull-secret-syncer-tt5mj"
Oct 30 04:44:32.114160 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.112387    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-multus-socket-dir-parent\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:44:32.114160 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.112400    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2a3eb4b-a128-4649-bf77-78df2705f147-cnibin\") pod \"multus-additional-cni-plugins-tnwc6\" (UID: \"a2a3eb4b-a128-4649-bf77-78df2705f147\") " pod="openshift-multus/multus-additional-cni-plugins-tnwc6"
Oct 30 04:44:32.114160 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.112424    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-log-socket\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:44:32.114160 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.112433    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/41a4ec4b-b407-4316-9238-dee6be432d3c-dbus\") pod \"global-pull-secret-syncer-tt5mj\" (UID: \"41a4ec4b-b407-4316-9238-dee6be432d3c\") " pod="kube-system/global-pull-secret-syncer-tt5mj"
Oct 30 04:44:32.114160 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.112448    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-lib-modules\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:44:32.114160 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.112470    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-log-socket\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:44:32.114160 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.112476    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2a3eb4b-a128-4649-bf77-78df2705f147-cnibin\") pod \"multus-additional-cni-plugins-tnwc6\" (UID: \"a2a3eb4b-a128-4649-bf77-78df2705f147\") " pod="openshift-multus/multus-additional-cni-plugins-tnwc6"
Oct 30 04:44:32.114160 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.112472    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-host\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:44:32.114160 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.112512    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-host\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:44:32.114160 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.112516    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/672524df-2f52-455b-9423-a58f53c31129-etc-selinux\") pod \"aws-ebs-csi-driver-node-vhsc7\" (UID: \"672524df-2f52-455b-9423-a58f53c31129\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhsc7"
Oct 30 04:44:32.114160 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.112536    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ff3a5b12-bdd2-47d1-9475-ee3b2bcba935-lib-modules\") pod \"tuned-qtb8v\" (UID: \"ff3a5b12-bdd2-47d1-9475-ee3b2bcba935\") " pod="openshift-cluster-node-tuning-operator/tuned-qtb8v"
Oct 30 04:44:32.114160 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.112543    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-run-systemd\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:44:32.114160 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.112554    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/672524df-2f52-455b-9423-a58f53c31129-etc-selinux\") pod \"aws-ebs-csi-driver-node-vhsc7\" (UID: \"672524df-2f52-455b-9423-a58f53c31129\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhsc7"
Oct 30 04:44:32.114160 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.112566    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-run-systemd\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:44:32.114160 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.112571    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/672524df-2f52-455b-9423-a58f53c31129-registration-dir\") pod \"aws-ebs-csi-driver-node-vhsc7\" (UID: \"672524df-2f52-455b-9423-a58f53c31129\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhsc7"
Oct 30 04:44:32.114160 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.112592    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-host-cni-netd\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:44:32.114160 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.112610    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-host-var-lib-cni-multus\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:44:32.114160 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.112666    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/70169887-cc72-4fb5-96b5-e09e260d7a8b-host-cni-netd\") pod \"ovnkube-node-v4bnv\" (UID: \"70169887-cc72-4fb5-96b5-e09e260d7a8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:44:32.114160 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.112680    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/672524df-2f52-455b-9423-a58f53c31129-registration-dir\") pod \"aws-ebs-csi-driver-node-vhsc7\" (UID: \"672524df-2f52-455b-9423-a58f53c31129\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vhsc7"
Oct 30 04:44:32.114160 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.112691    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bf49335f-7b8e-4212-8777-1575cd0929f1-host-var-lib-cni-multus\") pod \"multus-c9tpc\" (UID: \"bf49335f-7b8e-4212-8777-1575cd0929f1\") " pod="openshift-multus/multus-c9tpc"
Oct 30 04:44:32.200929 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.200880    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:32.269348 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.269313    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f66c46c7c-skmxs"]
Oct 30 04:44:32.397664 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.397579    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:32.473465 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.473438    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-f66c46c7c-skmxs"
Oct 30 04:44:32.483467 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.483434    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:32.549111 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:32.549091    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-bbk54"
Oct 30 04:44:33.296249 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:33.296223    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:44:34.153568 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:34.153541    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:44:34.153739 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:34.153732    5898 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 04:44:34.959376 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:34.959346    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-r5tdl"
Oct 30 04:44:35.747038 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:35.747006    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-95b457c5b-n92bg"
Oct 30 04:44:35.756644 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:35.756628    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7db77977b9-2f8qv"
Oct 30 04:44:36.469647 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:36.469609    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:36.574916 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:36.574866    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:36.658637 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:36.658589    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:37.636862 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:37.636818    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:37.723786 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:37.723757    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:41.172565 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:41.172528    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dgsdt"
Oct 30 04:44:42.047280 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:42.047199    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7db77977b9-2f8qv"
Oct 30 04:44:42.245537 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:42.245468    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:42.327040 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:42.326999    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:43.022684 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:43.022652    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6bpl4"
Oct 30 04:44:45.761395 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:45.761368    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7db77977b9-2f8qv"
Oct 30 04:44:45.765073 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:45.765051    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7db77977b9-2f8qv"
Oct 30 04:44:46.078929 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:46.078881    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:46.163052 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:46.163022    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:50.025002 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:50.024968    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:50.214827 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:50.214789    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:50.684100 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:50.684066    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:50.777606 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:50.777564    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:50.856847 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:50.856814    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:56.816313 ip-10-0-135-98 kubenswrapper[5898]: E1030 04:44:56.816276    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 04:44:57.237456 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:57.237419    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:57.329540 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:57.329513    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:57.973075 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:57.973035    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-f66c46c7c-skmxs" podUID="faa6a40d-2e9e-4348-a599-0df6f2d0177d" containerName="console" containerID="cri-o://51265ebd62788e2d4bb19965906472dc9a0d2e30f50ba8139ca61f25c48493e7" gracePeriod=15
Oct 30 04:44:58.220985 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:58.220962    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f66c46c7c-skmxs"
Oct 30 04:44:58.301429 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:58.301372    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/faa6a40d-2e9e-4348-a599-0df6f2d0177d-console-config\" (UniqueName: \"kubernetes.io/configmap/faa6a40d-2e9e-4348-a599-0df6f2d0177d-console-config\") pod \"faa6a40d-2e9e-4348-a599-0df6f2d0177d\" (UID: \"faa6a40d-2e9e-4348-a599-0df6f2d0177d\") "
Oct 30 04:44:58.301429 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:58.301406    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faa6a40d-2e9e-4348-a599-0df6f2d0177d-tmp\" (UniqueName: \"kubernetes.io/empty-dir/faa6a40d-2e9e-4348-a599-0df6f2d0177d-tmp\") pod \"faa6a40d-2e9e-4348-a599-0df6f2d0177d\" (UID: \"faa6a40d-2e9e-4348-a599-0df6f2d0177d\") "
Oct 30 04:44:58.301591 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:58.301452    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/faa6a40d-2e9e-4348-a599-0df6f2d0177d-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/faa6a40d-2e9e-4348-a599-0df6f2d0177d-oauth-serving-cert\") pod \"faa6a40d-2e9e-4348-a599-0df6f2d0177d\" (UID: \"faa6a40d-2e9e-4348-a599-0df6f2d0177d\") "
Oct 30 04:44:58.301591 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:58.301493    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/faa6a40d-2e9e-4348-a599-0df6f2d0177d-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/faa6a40d-2e9e-4348-a599-0df6f2d0177d-console-serving-cert\") pod \"faa6a40d-2e9e-4348-a599-0df6f2d0177d\" (UID: \"faa6a40d-2e9e-4348-a599-0df6f2d0177d\") "
Oct 30 04:44:58.301591 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:58.301518    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faa6a40d-2e9e-4348-a599-0df6f2d0177d-kube-api-access-p5q8q\" (UniqueName: \"kubernetes.io/projected/faa6a40d-2e9e-4348-a599-0df6f2d0177d-kube-api-access-p5q8q\") pod \"faa6a40d-2e9e-4348-a599-0df6f2d0177d\" (UID: \"faa6a40d-2e9e-4348-a599-0df6f2d0177d\") "
Oct 30 04:44:58.301591 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:58.301538    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/faa6a40d-2e9e-4348-a599-0df6f2d0177d-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/faa6a40d-2e9e-4348-a599-0df6f2d0177d-console-oauth-config\") pod \"faa6a40d-2e9e-4348-a599-0df6f2d0177d\" (UID: \"faa6a40d-2e9e-4348-a599-0df6f2d0177d\") "
Oct 30 04:44:58.301789 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:58.301635    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/faa6a40d-2e9e-4348-a599-0df6f2d0177d-service-ca\" (UniqueName: \"kubernetes.io/configmap/faa6a40d-2e9e-4348-a599-0df6f2d0177d-service-ca\") pod \"faa6a40d-2e9e-4348-a599-0df6f2d0177d\" (UID: \"faa6a40d-2e9e-4348-a599-0df6f2d0177d\") "
Oct 30 04:44:58.301847 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:58.301806    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faa6a40d-2e9e-4348-a599-0df6f2d0177d-tmp" pod "faa6a40d-2e9e-4348-a599-0df6f2d0177d" (UID: "faa6a40d-2e9e-4348-a599-0df6f2d0177d"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:44:58.301847 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:58.301823    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faa6a40d-2e9e-4348-a599-0df6f2d0177d-console-config" pod "faa6a40d-2e9e-4348-a599-0df6f2d0177d" (UID: "faa6a40d-2e9e-4348-a599-0df6f2d0177d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:44:58.301988 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:58.301881    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faa6a40d-2e9e-4348-a599-0df6f2d0177d-oauth-serving-cert" pod "faa6a40d-2e9e-4348-a599-0df6f2d0177d" (UID: "faa6a40d-2e9e-4348-a599-0df6f2d0177d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:44:58.302255 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:58.302231    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faa6a40d-2e9e-4348-a599-0df6f2d0177d-service-ca" pod "faa6a40d-2e9e-4348-a599-0df6f2d0177d" (UID: "faa6a40d-2e9e-4348-a599-0df6f2d0177d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:44:58.303849 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:58.303824    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/faa6a40d-2e9e-4348-a599-0df6f2d0177d-console-serving-cert" pod "faa6a40d-2e9e-4348-a599-0df6f2d0177d" (UID: "faa6a40d-2e9e-4348-a599-0df6f2d0177d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:44:58.303971 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:58.303892    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/faa6a40d-2e9e-4348-a599-0df6f2d0177d-console-oauth-config" pod "faa6a40d-2e9e-4348-a599-0df6f2d0177d" (UID: "faa6a40d-2e9e-4348-a599-0df6f2d0177d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:44:58.303971 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:58.303889    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faa6a40d-2e9e-4348-a599-0df6f2d0177d-kube-api-access-p5q8q" pod "faa6a40d-2e9e-4348-a599-0df6f2d0177d" (UID: "faa6a40d-2e9e-4348-a599-0df6f2d0177d"). InnerVolumeSpecName "kube-api-access-p5q8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:44:58.402822 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:58.402801    5898 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/faa6a40d-2e9e-4348-a599-0df6f2d0177d-oauth-serving-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 04:44:58.402822 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:58.402821    5898 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/faa6a40d-2e9e-4348-a599-0df6f2d0177d-console-serving-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 04:44:58.402957 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:58.402830    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5q8q\" (UniqueName: \"kubernetes.io/projected/faa6a40d-2e9e-4348-a599-0df6f2d0177d-kube-api-access-p5q8q\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 04:44:58.402957 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:58.402839    5898 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/faa6a40d-2e9e-4348-a599-0df6f2d0177d-console-oauth-config\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 04:44:58.402957 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:58.402848    5898 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/faa6a40d-2e9e-4348-a599-0df6f2d0177d-service-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 04:44:58.402957 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:58.402856    5898 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/faa6a40d-2e9e-4348-a599-0df6f2d0177d-console-config\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 04:44:58.402957 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:58.402864    5898 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/faa6a40d-2e9e-4348-a599-0df6f2d0177d-tmp\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 04:44:58.956396 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:58.956363    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-84b56d49-zl4cd"]
Oct 30 04:44:58.956695 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:58.956676    5898 cpu_manager.go:414] "containerMap: removing container" podUID="faa6a40d-2e9e-4348-a599-0df6f2d0177d" containerName="console"
Oct 30 04:44:58.956773 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:58.956696    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faa6a40d-2e9e-4348-a599-0df6f2d0177d" containerName="console"
Oct 30 04:44:58.956825 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:58.956780    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="faa6a40d-2e9e-4348-a599-0df6f2d0177d" containerName="console"
Oct 30 04:44:58.958729 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:58.958707    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84b56d49-zl4cd"
Oct 30 04:44:58.978811 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:58.978781    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84b56d49-zl4cd"]
Oct 30 04:44:59.007060 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:59.007039    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5-service-ca\") pod \"console-84b56d49-zl4cd\" (UID: \"10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5\") " pod="openshift-console/console-84b56d49-zl4cd"
Oct 30 04:44:59.007192 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:59.007070    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5-trusted-ca-bundle\") pod \"console-84b56d49-zl4cd\" (UID: \"10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5\") " pod="openshift-console/console-84b56d49-zl4cd"
Oct 30 04:44:59.007192 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:59.007090    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5-console-oauth-config\") pod \"console-84b56d49-zl4cd\" (UID: \"10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5\") " pod="openshift-console/console-84b56d49-zl4cd"
Oct 30 04:44:59.007192 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:59.007145    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5-console-serving-cert\") pod \"console-84b56d49-zl4cd\" (UID: \"10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5\") " pod="openshift-console/console-84b56d49-zl4cd"
Oct 30 04:44:59.007359 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:59.007290    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5-console-config\") pod \"console-84b56d49-zl4cd\" (UID: \"10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5\") " pod="openshift-console/console-84b56d49-zl4cd"
Oct 30 04:44:59.007399 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:59.007370    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x62pf\" (UniqueName: \"kubernetes.io/projected/10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5-kube-api-access-x62pf\") pod \"console-84b56d49-zl4cd\" (UID: \"10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5\") " pod="openshift-console/console-84b56d49-zl4cd"
Oct 30 04:44:59.007399 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:59.007392    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5-tmp\") pod \"console-84b56d49-zl4cd\" (UID: \"10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5\") " pod="openshift-console/console-84b56d49-zl4cd"
Oct 30 04:44:59.007481 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:59.007411    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5-oauth-serving-cert\") pod \"console-84b56d49-zl4cd\" (UID: \"10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5\") " pod="openshift-console/console-84b56d49-zl4cd"
Oct 30 04:44:59.038671 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:59.037810    5898 generic.go:356] "Generic (PLEG): container finished" podID="faa6a40d-2e9e-4348-a599-0df6f2d0177d" containerID="51265ebd62788e2d4bb19965906472dc9a0d2e30f50ba8139ca61f25c48493e7" exitCode=0
Oct 30 04:44:59.038671 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:59.037957    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f66c46c7c-skmxs" event={"ID":"faa6a40d-2e9e-4348-a599-0df6f2d0177d","Type":"ContainerDied","Data":"51265ebd62788e2d4bb19965906472dc9a0d2e30f50ba8139ca61f25c48493e7"}
Oct 30 04:44:59.038671 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:59.038036    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f66c46c7c-skmxs" event={"ID":"faa6a40d-2e9e-4348-a599-0df6f2d0177d","Type":"ContainerDied","Data":"1c2c7e85e42f36c1dd3132a5fc87e0b98b6d22ea9dec0a9c5cd39844c3050fa9"}
Oct 30 04:44:59.038671 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:59.038066    5898 scope.go:122] "RemoveContainer" containerID="51265ebd62788e2d4bb19965906472dc9a0d2e30f50ba8139ca61f25c48493e7"
Oct 30 04:44:59.038671 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:59.038251    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f66c46c7c-skmxs"
Oct 30 04:44:59.046276 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:59.046255    5898 scope.go:122] "RemoveContainer" containerID="51265ebd62788e2d4bb19965906472dc9a0d2e30f50ba8139ca61f25c48493e7"
Oct 30 04:44:59.046538 ip-10-0-135-98 kubenswrapper[5898]: E1030 04:44:59.046510    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51265ebd62788e2d4bb19965906472dc9a0d2e30f50ba8139ca61f25c48493e7\": container with ID starting with 51265ebd62788e2d4bb19965906472dc9a0d2e30f50ba8139ca61f25c48493e7 not found: ID does not exist" containerID="51265ebd62788e2d4bb19965906472dc9a0d2e30f50ba8139ca61f25c48493e7"
Oct 30 04:44:59.046629 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:59.046547    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51265ebd62788e2d4bb19965906472dc9a0d2e30f50ba8139ca61f25c48493e7"} err="failed to get container status \"51265ebd62788e2d4bb19965906472dc9a0d2e30f50ba8139ca61f25c48493e7\": rpc error: code = NotFound desc = could not find container \"51265ebd62788e2d4bb19965906472dc9a0d2e30f50ba8139ca61f25c48493e7\": container with ID starting with 51265ebd62788e2d4bb19965906472dc9a0d2e30f50ba8139ca61f25c48493e7 not found: ID does not exist"
Oct 30 04:44:59.071859 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:59.071834    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:59.084544 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:59.084501    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f66c46c7c-skmxs"]
Oct 30 04:44:59.088224 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:59.088198    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-f66c46c7c-skmxs"]
Oct 30 04:44:59.108139 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:59.108116    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5-tmp\") pod \"console-84b56d49-zl4cd\" (UID: \"10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5\") " pod="openshift-console/console-84b56d49-zl4cd"
Oct 30 04:44:59.108224 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:59.108143    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5-oauth-serving-cert\") pod \"console-84b56d49-zl4cd\" (UID: \"10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5\") " pod="openshift-console/console-84b56d49-zl4cd"
Oct 30 04:44:59.108224 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:59.108162    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5-service-ca\") pod \"console-84b56d49-zl4cd\" (UID: \"10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5\") " pod="openshift-console/console-84b56d49-zl4cd"
Oct 30 04:44:59.108224 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:59.108189    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5-trusted-ca-bundle\") pod \"console-84b56d49-zl4cd\" (UID: \"10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5\") " pod="openshift-console/console-84b56d49-zl4cd"
Oct 30 04:44:59.108224 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:59.108212    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5-console-oauth-config\") pod \"console-84b56d49-zl4cd\" (UID: \"10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5\") " pod="openshift-console/console-84b56d49-zl4cd"
Oct 30 04:44:59.108406 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:59.108246    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5-console-serving-cert\") pod \"console-84b56d49-zl4cd\" (UID: \"10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5\") " pod="openshift-console/console-84b56d49-zl4cd"
Oct 30 04:44:59.108406 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:59.108282    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5-console-config\") pod \"console-84b56d49-zl4cd\" (UID: \"10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5\") " pod="openshift-console/console-84b56d49-zl4cd"
Oct 30 04:44:59.108876 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:59.108562    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x62pf\" (UniqueName: \"kubernetes.io/projected/10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5-kube-api-access-x62pf\") pod \"console-84b56d49-zl4cd\" (UID: \"10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5\") " pod="openshift-console/console-84b56d49-zl4cd"
Oct 30 04:44:59.108876 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:59.108571    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5-tmp\") pod \"console-84b56d49-zl4cd\" (UID: \"10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5\") " pod="openshift-console/console-84b56d49-zl4cd"
Oct 30 04:44:59.108876 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:59.108680    5898 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 04:44:59.109108 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:59.108971    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5-service-ca\") pod \"console-84b56d49-zl4cd\" (UID: \"10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5\") " pod="openshift-console/console-84b56d49-zl4cd"
Oct 30 04:44:59.109108 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:59.108978    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5-oauth-serving-cert\") pod \"console-84b56d49-zl4cd\" (UID: \"10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5\") " pod="openshift-console/console-84b56d49-zl4cd"
Oct 30 04:44:59.109211 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:59.109164    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5-trusted-ca-bundle\") pod \"console-84b56d49-zl4cd\" (UID: \"10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5\") " pod="openshift-console/console-84b56d49-zl4cd"
Oct 30 04:44:59.109211 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:59.109175    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5-console-config\") pod \"console-84b56d49-zl4cd\" (UID: \"10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5\") " pod="openshift-console/console-84b56d49-zl4cd"
Oct 30 04:44:59.110795 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:59.110773    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5-console-oauth-config\") pod \"console-84b56d49-zl4cd\" (UID: \"10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5\") " pod="openshift-console/console-84b56d49-zl4cd"
Oct 30 04:44:59.111028 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:59.111008    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5-console-serving-cert\") pod \"console-84b56d49-zl4cd\" (UID: \"10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5\") " pod="openshift-console/console-84b56d49-zl4cd"
Oct 30 04:44:59.126380 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:59.126356    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x62pf\" (UniqueName: \"kubernetes.io/projected/10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5-kube-api-access-x62pf\") pod \"console-84b56d49-zl4cd\" (UID: \"10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5\") " pod="openshift-console/console-84b56d49-zl4cd"
Oct 30 04:44:59.268015 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:59.267937    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84b56d49-zl4cd"
Oct 30 04:44:59.370313 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:59.370284    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:59.470642 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:59.470561    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:44:59.556133 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:44:59.556051    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84b56d49-zl4cd"]
Oct 30 04:44:59.560651 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:44:59.560623    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10b0b5a4_7f6b_4b56_b5c7_08713e4ba0d5.slice/crio-2e1a19aef02f30e1a6e940461b219fa90933a50fd68eb1776bdf126c3a035bf4 WatchSource:0}: Error finding container 2e1a19aef02f30e1a6e940461b219fa90933a50fd68eb1776bdf126c3a035bf4: Status 404 returned error can't find the container with id 2e1a19aef02f30e1a6e940461b219fa90933a50fd68eb1776bdf126c3a035bf4
Oct 30 04:45:00.042551 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:00.042514    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84b56d49-zl4cd" event={"ID":"10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5","Type":"ContainerStarted","Data":"1811c89ea38f156b105f74df30f46e24aa87332f2352080ea455efbef7092310"}
Oct 30 04:45:00.043017 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:00.042569    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84b56d49-zl4cd" event={"ID":"10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5","Type":"ContainerStarted","Data":"2e1a19aef02f30e1a6e940461b219fa90933a50fd68eb1776bdf126c3a035bf4"}
Oct 30 04:45:00.091100 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:00.091049    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-84b56d49-zl4cd" podStartSLOduration=2.091035624 podStartE2EDuration="2.091035624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:44:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:45:00.079458306 +0000 UTC m=+29.303278206" watchObservedRunningTime="2026-10-30 04:45:00.091035624 +0000 UTC m=+29.314855518"
Oct 30 04:45:00.264529 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:00.264494    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v4bnv"
Oct 30 04:45:00.887046 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:00.887015    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="faa6a40d-2e9e-4348-a599-0df6f2d0177d" path="/var/lib/kubelet/pods/faa6a40d-2e9e-4348-a599-0df6f2d0177d/volumes"
Oct 30 04:45:01.175765 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:01.175694    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-dgsdt"
Oct 30 04:45:09.268378 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:09.268342    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-84b56d49-zl4cd"
Oct 30 04:45:09.268877 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:09.268395    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-84b56d49-zl4cd"
Oct 30 04:45:09.273079 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:09.273059    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-84b56d49-zl4cd"
Oct 30 04:45:09.509497 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:09.509462    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:45:10.078885 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:10.078854    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-84b56d49-zl4cd"
Oct 30 04:45:10.138942 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:10.138896    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7db77977b9-2f8qv"]
Oct 30 04:45:26.815815 ip-10-0-135-98 kubenswrapper[5898]: E1030 04:45:26.815777    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 04:45:35.160852 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:35.160787    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7db77977b9-2f8qv" podUID="47f03fac-07ba-4d50-9336-c5af8b4fbcc3" containerName="console" containerID="cri-o://b70a5072438ea0d797b5909d950bb8908f13d596a25fa233a362e066026c2bb2" gracePeriod=15
Oct 30 04:45:35.392539 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:35.392515    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7db77977b9-2f8qv"
Oct 30 04:45:35.461749 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:35.461686    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-tmp\" (UniqueName: \"kubernetes.io/empty-dir/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-tmp\") pod \"47f03fac-07ba-4d50-9336-c5af8b4fbcc3\" (UID: \"47f03fac-07ba-4d50-9336-c5af8b4fbcc3\") "
Oct 30 04:45:35.461749 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:35.461724    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-console-config\" (UniqueName: \"kubernetes.io/configmap/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-console-config\") pod \"47f03fac-07ba-4d50-9336-c5af8b4fbcc3\" (UID: \"47f03fac-07ba-4d50-9336-c5af8b4fbcc3\") "
Oct 30 04:45:35.461933 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:35.461751    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-console-oauth-config\") pod \"47f03fac-07ba-4d50-9336-c5af8b4fbcc3\" (UID: \"47f03fac-07ba-4d50-9336-c5af8b4fbcc3\") "
Oct 30 04:45:35.461933 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:35.461857    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-console-serving-cert\") pod \"47f03fac-07ba-4d50-9336-c5af8b4fbcc3\" (UID: \"47f03fac-07ba-4d50-9336-c5af8b4fbcc3\") "
Oct 30 04:45:35.461933 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:35.461890    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-service-ca\" (UniqueName: \"kubernetes.io/configmap/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-service-ca\") pod \"47f03fac-07ba-4d50-9336-c5af8b4fbcc3\" (UID: \"47f03fac-07ba-4d50-9336-c5af8b4fbcc3\") "
Oct 30 04:45:35.462089 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:35.461939    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-oauth-serving-cert\") pod \"47f03fac-07ba-4d50-9336-c5af8b4fbcc3\" (UID: \"47f03fac-07ba-4d50-9336-c5af8b4fbcc3\") "
Oct 30 04:45:35.462089 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:35.461980    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-trusted-ca-bundle\") pod \"47f03fac-07ba-4d50-9336-c5af8b4fbcc3\" (UID: \"47f03fac-07ba-4d50-9336-c5af8b4fbcc3\") "
Oct 30 04:45:35.462089 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:35.462020    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-kube-api-access-z6rb2\" (UniqueName: \"kubernetes.io/projected/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-kube-api-access-z6rb2\") pod \"47f03fac-07ba-4d50-9336-c5af8b4fbcc3\" (UID: \"47f03fac-07ba-4d50-9336-c5af8b4fbcc3\") "
Oct 30 04:45:35.462089 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:35.461970    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-tmp" pod "47f03fac-07ba-4d50-9336-c5af8b4fbcc3" (UID: "47f03fac-07ba-4d50-9336-c5af8b4fbcc3"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:45:35.462295 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:35.462094    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-console-config" pod "47f03fac-07ba-4d50-9336-c5af8b4fbcc3" (UID: "47f03fac-07ba-4d50-9336-c5af8b4fbcc3"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:45:35.462356 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:35.462321    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-service-ca" pod "47f03fac-07ba-4d50-9336-c5af8b4fbcc3" (UID: "47f03fac-07ba-4d50-9336-c5af8b4fbcc3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:45:35.462415 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:35.462363    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-oauth-serving-cert" pod "47f03fac-07ba-4d50-9336-c5af8b4fbcc3" (UID: "47f03fac-07ba-4d50-9336-c5af8b4fbcc3"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:45:35.462415 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:35.462378    5898 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-tmp\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 04:45:35.462415 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:35.462394    5898 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-console-config\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 04:45:35.462560 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:35.462430    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-trusted-ca-bundle" pod "47f03fac-07ba-4d50-9336-c5af8b4fbcc3" (UID: "47f03fac-07ba-4d50-9336-c5af8b4fbcc3"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:45:35.463934 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:35.463894    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-console-oauth-config" pod "47f03fac-07ba-4d50-9336-c5af8b4fbcc3" (UID: "47f03fac-07ba-4d50-9336-c5af8b4fbcc3"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:45:35.464050 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:35.464018    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-console-serving-cert" pod "47f03fac-07ba-4d50-9336-c5af8b4fbcc3" (UID: "47f03fac-07ba-4d50-9336-c5af8b4fbcc3"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:45:35.464267 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:35.464250    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-kube-api-access-z6rb2" pod "47f03fac-07ba-4d50-9336-c5af8b4fbcc3" (UID: "47f03fac-07ba-4d50-9336-c5af8b4fbcc3"). InnerVolumeSpecName "kube-api-access-z6rb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:45:35.563422 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:35.563394    5898 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-console-serving-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 04:45:35.563422 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:35.563418    5898 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-service-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 04:45:35.563422 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:35.563427    5898 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-oauth-serving-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 04:45:35.563628 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:35.563438    5898 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-trusted-ca-bundle\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 04:45:35.563628 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:35.563447    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6rb2\" (UniqueName: \"kubernetes.io/projected/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-kube-api-access-z6rb2\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 04:45:35.563628 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:35.563459    5898 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/47f03fac-07ba-4d50-9336-c5af8b4fbcc3-console-oauth-config\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 04:45:36.154579 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:36.154547    5898 generic.go:356] "Generic (PLEG): container finished" podID="47f03fac-07ba-4d50-9336-c5af8b4fbcc3" containerID="b70a5072438ea0d797b5909d950bb8908f13d596a25fa233a362e066026c2bb2" exitCode=0
Oct 30 04:45:36.154765 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:36.154597    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7db77977b9-2f8qv"
Oct 30 04:45:36.154765 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:36.154618    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7db77977b9-2f8qv" event={"ID":"47f03fac-07ba-4d50-9336-c5af8b4fbcc3","Type":"ContainerDied","Data":"b70a5072438ea0d797b5909d950bb8908f13d596a25fa233a362e066026c2bb2"}
Oct 30 04:45:36.154765 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:36.154661    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7db77977b9-2f8qv" event={"ID":"47f03fac-07ba-4d50-9336-c5af8b4fbcc3","Type":"ContainerDied","Data":"55e9f36fd3d87ecd971b3336a06458ceae8c710341869364e8912f232c844aac"}
Oct 30 04:45:36.154765 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:36.154687    5898 scope.go:122] "RemoveContainer" containerID="b70a5072438ea0d797b5909d950bb8908f13d596a25fa233a362e066026c2bb2"
Oct 30 04:45:36.162757 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:36.162645    5898 scope.go:122] "RemoveContainer" containerID="b70a5072438ea0d797b5909d950bb8908f13d596a25fa233a362e066026c2bb2"
Oct 30 04:45:36.163047 ip-10-0-135-98 kubenswrapper[5898]: E1030 04:45:36.162958    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b70a5072438ea0d797b5909d950bb8908f13d596a25fa233a362e066026c2bb2\": container with ID starting with b70a5072438ea0d797b5909d950bb8908f13d596a25fa233a362e066026c2bb2 not found: ID does not exist" containerID="b70a5072438ea0d797b5909d950bb8908f13d596a25fa233a362e066026c2bb2"
Oct 30 04:45:36.163047 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:36.162987    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b70a5072438ea0d797b5909d950bb8908f13d596a25fa233a362e066026c2bb2"} err="failed to get container status \"b70a5072438ea0d797b5909d950bb8908f13d596a25fa233a362e066026c2bb2\": rpc error: code = NotFound desc = could not find container \"b70a5072438ea0d797b5909d950bb8908f13d596a25fa233a362e066026c2bb2\": container with ID starting with b70a5072438ea0d797b5909d950bb8908f13d596a25fa233a362e066026c2bb2 not found: ID does not exist"
Oct 30 04:45:36.202999 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:36.202977    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7db77977b9-2f8qv"]
Oct 30 04:45:36.214414 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:36.214396    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7db77977b9-2f8qv"]
Oct 30 04:45:36.610091 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:36.610057    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:45:36.887116 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:36.887047    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47f03fac-07ba-4d50-9336-c5af8b4fbcc3" path="/var/lib/kubelet/pods/47f03fac-07ba-4d50-9336-c5af8b4fbcc3/volumes"
Oct 30 04:45:45.891949 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:45.891891    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:45:45.976555 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:45.976523    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:45:47.236579 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:47.236549    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7ddddfc654-x4l4p"]
Oct 30 04:45:47.236970 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:47.236780    5898 cpu_manager.go:414] "containerMap: removing container" podUID="47f03fac-07ba-4d50-9336-c5af8b4fbcc3" containerName="console"
Oct 30 04:45:47.236970 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:47.236790    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47f03fac-07ba-4d50-9336-c5af8b4fbcc3" containerName="console"
Oct 30 04:45:47.236970 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:47.236849    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="47f03fac-07ba-4d50-9336-c5af8b4fbcc3" containerName="console"
Oct 30 04:45:47.238638 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:47.238624    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7ddddfc654-x4l4p"
Oct 30 04:45:47.264177 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:47.264154    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7ddddfc654-x4l4p"]
Oct 30 04:45:47.345875 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:47.345850    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1fb50298-92f0-4411-a3dd-ddef516b4314-console-oauth-config\") pod \"console-7ddddfc654-x4l4p\" (UID: \"1fb50298-92f0-4411-a3dd-ddef516b4314\") " pod="openshift-console/console-7ddddfc654-x4l4p"
Oct 30 04:45:47.346040 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:47.345886    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1fb50298-92f0-4411-a3dd-ddef516b4314-tmp\") pod \"console-7ddddfc654-x4l4p\" (UID: \"1fb50298-92f0-4411-a3dd-ddef516b4314\") " pod="openshift-console/console-7ddddfc654-x4l4p"
Oct 30 04:45:47.346040 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:47.345982    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1fb50298-92f0-4411-a3dd-ddef516b4314-console-serving-cert\") pod \"console-7ddddfc654-x4l4p\" (UID: \"1fb50298-92f0-4411-a3dd-ddef516b4314\") " pod="openshift-console/console-7ddddfc654-x4l4p"
Oct 30 04:45:47.346040 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:47.346024    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1fb50298-92f0-4411-a3dd-ddef516b4314-service-ca\") pod \"console-7ddddfc654-x4l4p\" (UID: \"1fb50298-92f0-4411-a3dd-ddef516b4314\") " pod="openshift-console/console-7ddddfc654-x4l4p"
Oct 30 04:45:47.346171 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:47.346053    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1fb50298-92f0-4411-a3dd-ddef516b4314-console-config\") pod \"console-7ddddfc654-x4l4p\" (UID: \"1fb50298-92f0-4411-a3dd-ddef516b4314\") " pod="openshift-console/console-7ddddfc654-x4l4p"
Oct 30 04:45:47.346171 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:47.346109    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1fb50298-92f0-4411-a3dd-ddef516b4314-trusted-ca-bundle\") pod \"console-7ddddfc654-x4l4p\" (UID: \"1fb50298-92f0-4411-a3dd-ddef516b4314\") " pod="openshift-console/console-7ddddfc654-x4l4p"
Oct 30 04:45:47.346171 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:47.346143    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1fb50298-92f0-4411-a3dd-ddef516b4314-oauth-serving-cert\") pod \"console-7ddddfc654-x4l4p\" (UID: \"1fb50298-92f0-4411-a3dd-ddef516b4314\") " pod="openshift-console/console-7ddddfc654-x4l4p"
Oct 30 04:45:47.346267 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:47.346212    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnfrd\" (UniqueName: \"kubernetes.io/projected/1fb50298-92f0-4411-a3dd-ddef516b4314-kube-api-access-tnfrd\") pod \"console-7ddddfc654-x4l4p\" (UID: \"1fb50298-92f0-4411-a3dd-ddef516b4314\") " pod="openshift-console/console-7ddddfc654-x4l4p"
Oct 30 04:45:47.351737 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:47.351714    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:45:47.446569 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:47.446536    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1fb50298-92f0-4411-a3dd-ddef516b4314-service-ca\") pod \"console-7ddddfc654-x4l4p\" (UID: \"1fb50298-92f0-4411-a3dd-ddef516b4314\") " pod="openshift-console/console-7ddddfc654-x4l4p"
Oct 30 04:45:47.446569 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:47.446577    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1fb50298-92f0-4411-a3dd-ddef516b4314-console-config\") pod \"console-7ddddfc654-x4l4p\" (UID: \"1fb50298-92f0-4411-a3dd-ddef516b4314\") " pod="openshift-console/console-7ddddfc654-x4l4p"
Oct 30 04:45:47.446826 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:47.446595    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1fb50298-92f0-4411-a3dd-ddef516b4314-trusted-ca-bundle\") pod \"console-7ddddfc654-x4l4p\" (UID: \"1fb50298-92f0-4411-a3dd-ddef516b4314\") " pod="openshift-console/console-7ddddfc654-x4l4p"
Oct 30 04:45:47.446826 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:47.446657    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1fb50298-92f0-4411-a3dd-ddef516b4314-oauth-serving-cert\") pod \"console-7ddddfc654-x4l4p\" (UID: \"1fb50298-92f0-4411-a3dd-ddef516b4314\") " pod="openshift-console/console-7ddddfc654-x4l4p"
Oct 30 04:45:47.446826 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:47.446696    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnfrd\" (UniqueName: \"kubernetes.io/projected/1fb50298-92f0-4411-a3dd-ddef516b4314-kube-api-access-tnfrd\") pod \"console-7ddddfc654-x4l4p\" (UID: \"1fb50298-92f0-4411-a3dd-ddef516b4314\") " pod="openshift-console/console-7ddddfc654-x4l4p"
Oct 30 04:45:47.446826 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:47.446743    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1fb50298-92f0-4411-a3dd-ddef516b4314-console-oauth-config\") pod \"console-7ddddfc654-x4l4p\" (UID: \"1fb50298-92f0-4411-a3dd-ddef516b4314\") " pod="openshift-console/console-7ddddfc654-x4l4p"
Oct 30 04:45:47.446826 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:47.446766    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1fb50298-92f0-4411-a3dd-ddef516b4314-tmp\") pod \"console-7ddddfc654-x4l4p\" (UID: \"1fb50298-92f0-4411-a3dd-ddef516b4314\") " pod="openshift-console/console-7ddddfc654-x4l4p"
Oct 30 04:45:47.446826 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:47.446801    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1fb50298-92f0-4411-a3dd-ddef516b4314-console-serving-cert\") pod \"console-7ddddfc654-x4l4p\" (UID: \"1fb50298-92f0-4411-a3dd-ddef516b4314\") " pod="openshift-console/console-7ddddfc654-x4l4p"
Oct 30 04:45:47.447364 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:47.447343    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1fb50298-92f0-4411-a3dd-ddef516b4314-tmp\") pod \"console-7ddddfc654-x4l4p\" (UID: \"1fb50298-92f0-4411-a3dd-ddef516b4314\") " pod="openshift-console/console-7ddddfc654-x4l4p"
Oct 30 04:45:47.447513 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:47.447368    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1fb50298-92f0-4411-a3dd-ddef516b4314-console-config\") pod \"console-7ddddfc654-x4l4p\" (UID: \"1fb50298-92f0-4411-a3dd-ddef516b4314\") " pod="openshift-console/console-7ddddfc654-x4l4p"
Oct 30 04:45:47.447615 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:47.447536    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1fb50298-92f0-4411-a3dd-ddef516b4314-trusted-ca-bundle\") pod \"console-7ddddfc654-x4l4p\" (UID: \"1fb50298-92f0-4411-a3dd-ddef516b4314\") " pod="openshift-console/console-7ddddfc654-x4l4p"
Oct 30 04:45:47.447749 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:47.447681    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1fb50298-92f0-4411-a3dd-ddef516b4314-oauth-serving-cert\") pod \"console-7ddddfc654-x4l4p\" (UID: \"1fb50298-92f0-4411-a3dd-ddef516b4314\") " pod="openshift-console/console-7ddddfc654-x4l4p"
Oct 30 04:45:47.447749 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:47.447732    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1fb50298-92f0-4411-a3dd-ddef516b4314-service-ca\") pod \"console-7ddddfc654-x4l4p\" (UID: \"1fb50298-92f0-4411-a3dd-ddef516b4314\") " pod="openshift-console/console-7ddddfc654-x4l4p"
Oct 30 04:45:47.449398 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:47.449373    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1fb50298-92f0-4411-a3dd-ddef516b4314-console-oauth-config\") pod \"console-7ddddfc654-x4l4p\" (UID: \"1fb50298-92f0-4411-a3dd-ddef516b4314\") " pod="openshift-console/console-7ddddfc654-x4l4p"
Oct 30 04:45:47.449542 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:47.449528    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1fb50298-92f0-4411-a3dd-ddef516b4314-console-serving-cert\") pod \"console-7ddddfc654-x4l4p\" (UID: \"1fb50298-92f0-4411-a3dd-ddef516b4314\") " pod="openshift-console/console-7ddddfc654-x4l4p"
Oct 30 04:45:47.452366 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:47.452335    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:45:47.465068 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:47.465047    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnfrd\" (UniqueName: \"kubernetes.io/projected/1fb50298-92f0-4411-a3dd-ddef516b4314-kube-api-access-tnfrd\") pod \"console-7ddddfc654-x4l4p\" (UID: \"1fb50298-92f0-4411-a3dd-ddef516b4314\") " pod="openshift-console/console-7ddddfc654-x4l4p"
Oct 30 04:45:47.548305 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:47.548232    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7ddddfc654-x4l4p"
Oct 30 04:45:47.662574 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:47.662538    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:45:47.746408 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:47.746379    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:45:47.851433 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:47.851403    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7ddddfc654-x4l4p"]
Oct 30 04:45:47.855026 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:45:47.855003    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fb50298_92f0_4411_a3dd_ddef516b4314.slice/crio-534a9d0ae0d36cac319c7445ffd499edbe05528aacf18669d564d212a6312ce7 WatchSource:0}: Error finding container 534a9d0ae0d36cac319c7445ffd499edbe05528aacf18669d564d212a6312ce7: Status 404 returned error can't find the container with id 534a9d0ae0d36cac319c7445ffd499edbe05528aacf18669d564d212a6312ce7
Oct 30 04:45:48.195865 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:48.195778    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7ddddfc654-x4l4p" event={"ID":"1fb50298-92f0-4411-a3dd-ddef516b4314","Type":"ContainerStarted","Data":"e3e9691a1b23ca5ddb4523a794a27a17a3d4585cb1ba3551af3aaa9eb1693246"}
Oct 30 04:45:48.195865 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:48.195826    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7ddddfc654-x4l4p" event={"ID":"1fb50298-92f0-4411-a3dd-ddef516b4314","Type":"ContainerStarted","Data":"534a9d0ae0d36cac319c7445ffd499edbe05528aacf18669d564d212a6312ce7"}
Oct 30 04:45:48.244035 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:48.243994    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7ddddfc654-x4l4p" podStartSLOduration=1.243981212 podStartE2EDuration="1.243981212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:45:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:45:48.232290208 +0000 UTC m=+77.456110103" watchObservedRunningTime="2026-10-30 04:45:48.243981212 +0000 UTC m=+77.467801107"
Oct 30 04:45:56.815781 ip-10-0-135-98 kubenswrapper[5898]: E1030 04:45:56.815745    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 04:45:57.548605 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:57.548561    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7ddddfc654-x4l4p"
Oct 30 04:45:57.548816 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:57.548632    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7ddddfc654-x4l4p"
Oct 30 04:45:57.553464 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:57.553439    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7ddddfc654-x4l4p"
Oct 30 04:45:58.241329 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:58.241300    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7ddddfc654-x4l4p"
Oct 30 04:45:58.310255 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:58.310225    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84b56d49-zl4cd"]
Oct 30 04:45:58.573565 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:58.573533    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:45:58.663018 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:45:58.662965    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:46:12.260929 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:46:12.260804    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:46:12.352688 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:46:12.352659    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:46:12.443216 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:46:12.443188    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:46:23.329781 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:46:23.329725    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-84b56d49-zl4cd" podUID="10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5" containerName="console" containerID="cri-o://1811c89ea38f156b105f74df30f46e24aa87332f2352080ea455efbef7092310" gracePeriod=15
Oct 30 04:46:23.561380 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:46:23.561353    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84b56d49-zl4cd"
Oct 30 04:46:23.697682 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:46:23.697604    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5-console-serving-cert\") pod \"10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5\" (UID: \"10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5\") "
Oct 30 04:46:23.697682 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:46:23.697652    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5-service-ca\" (UniqueName: \"kubernetes.io/configmap/10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5-service-ca\") pod \"10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5\" (UID: \"10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5\") "
Oct 30 04:46:23.697682 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:46:23.697670    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5-trusted-ca-bundle\") pod \"10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5\" (UID: \"10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5\") "
Oct 30 04:46:23.697969 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:46:23.697708    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5-tmp\" (UniqueName: \"kubernetes.io/empty-dir/10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5-tmp\") pod \"10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5\" (UID: \"10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5\") "
Oct 30 04:46:23.697969 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:46:23.697731    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5-console-config\" (UniqueName: \"kubernetes.io/configmap/10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5-console-config\") pod \"10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5\" (UID: \"10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5\") "
Oct 30 04:46:23.697969 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:46:23.697767    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5-console-oauth-config\") pod \"10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5\" (UID: \"10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5\") "
Oct 30 04:46:23.697969 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:46:23.697793    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5-oauth-serving-cert\") pod \"10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5\" (UID: \"10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5\") "
Oct 30 04:46:23.697969 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:46:23.697830    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5-kube-api-access-x62pf\" (UniqueName: \"kubernetes.io/projected/10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5-kube-api-access-x62pf\") pod \"10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5\" (UID: \"10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5\") "
Oct 30 04:46:23.698211 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:46:23.698082    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5-service-ca" pod "10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5" (UID: "10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:46:23.698211 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:46:23.698115    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5-tmp" pod "10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5" (UID: "10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:46:23.698211 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:46:23.698134    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5-trusted-ca-bundle" pod "10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5" (UID: "10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:46:23.698324 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:46:23.698299    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5-console-config" pod "10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5" (UID: "10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:46:23.698360 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:46:23.698336    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5-oauth-serving-cert" pod "10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5" (UID: "10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:46:23.700248 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:46:23.700226    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5-kube-api-access-x62pf" pod "10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5" (UID: "10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5"). InnerVolumeSpecName "kube-api-access-x62pf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:46:23.700248 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:46:23.700229    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5-console-serving-cert" pod "10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5" (UID: "10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:46:23.700385 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:46:23.700251    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5-console-oauth-config" pod "10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5" (UID: "10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:46:23.798527 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:46:23.798496    5898 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5-console-config\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 04:46:23.798527 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:46:23.798523    5898 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5-console-oauth-config\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 04:46:23.798527 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:46:23.798534    5898 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5-oauth-serving-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 04:46:23.798704 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:46:23.798543    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x62pf\" (UniqueName: \"kubernetes.io/projected/10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5-kube-api-access-x62pf\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 04:46:23.798704 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:46:23.798554    5898 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5-console-serving-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 04:46:23.798704 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:46:23.798562    5898 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5-service-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 04:46:23.798704 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:46:23.798571    5898 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5-trusted-ca-bundle\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 04:46:23.798704 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:46:23.798580    5898 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5-tmp\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 04:46:24.317947 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:46:24.317889    5898 generic.go:356] "Generic (PLEG): container finished" podID="10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5" containerID="1811c89ea38f156b105f74df30f46e24aa87332f2352080ea455efbef7092310" exitCode=0
Oct 30 04:46:24.318121 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:46:24.317959    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84b56d49-zl4cd"
Oct 30 04:46:24.318121 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:46:24.317956    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84b56d49-zl4cd" event={"ID":"10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5","Type":"ContainerDied","Data":"1811c89ea38f156b105f74df30f46e24aa87332f2352080ea455efbef7092310"}
Oct 30 04:46:24.318121 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:46:24.318078    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84b56d49-zl4cd" event={"ID":"10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5","Type":"ContainerDied","Data":"2e1a19aef02f30e1a6e940461b219fa90933a50fd68eb1776bdf126c3a035bf4"}
Oct 30 04:46:24.318121 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:46:24.318103    5898 scope.go:122] "RemoveContainer" containerID="1811c89ea38f156b105f74df30f46e24aa87332f2352080ea455efbef7092310"
Oct 30 04:46:24.326653 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:46:24.326637    5898 scope.go:122] "RemoveContainer" containerID="1811c89ea38f156b105f74df30f46e24aa87332f2352080ea455efbef7092310"
Oct 30 04:46:24.326921 ip-10-0-135-98 kubenswrapper[5898]: E1030 04:46:24.326887    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1811c89ea38f156b105f74df30f46e24aa87332f2352080ea455efbef7092310\": container with ID starting with 1811c89ea38f156b105f74df30f46e24aa87332f2352080ea455efbef7092310 not found: ID does not exist" containerID="1811c89ea38f156b105f74df30f46e24aa87332f2352080ea455efbef7092310"
Oct 30 04:46:24.326971 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:46:24.326933    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1811c89ea38f156b105f74df30f46e24aa87332f2352080ea455efbef7092310"} err="failed to get container status \"1811c89ea38f156b105f74df30f46e24aa87332f2352080ea455efbef7092310\": rpc error: code = NotFound desc = could not find container \"1811c89ea38f156b105f74df30f46e24aa87332f2352080ea455efbef7092310\": container with ID starting with 1811c89ea38f156b105f74df30f46e24aa87332f2352080ea455efbef7092310 not found: ID does not exist"
Oct 30 04:46:24.365745 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:46:24.365720    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84b56d49-zl4cd"]
Oct 30 04:46:24.377175 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:46:24.377156    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-84b56d49-zl4cd"]
Oct 30 04:46:24.889408 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:46:24.889370    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5" path="/var/lib/kubelet/pods/10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5/volumes"
Oct 30 04:46:26.815823 ip-10-0-135-98 kubenswrapper[5898]: E1030 04:46:26.815781    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 04:46:36.711963 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:46:36.711922    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:46:41.575581 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:46:41.575494    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:46:56.239217 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:46:56.239175    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:46:56.816695 ip-10-0-135-98 kubenswrapper[5898]: E1030 04:46:56.816657    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 04:47:26.816522 ip-10-0-135-98 kubenswrapper[5898]: E1030 04:47:26.816485    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 04:47:36.836429 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:47:36.836397    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:47:51.014631 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:47:51.014551    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:47:51.120956 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:47:51.120923    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:47:51.916434 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:47:51.916405    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv8xdv"]
Oct 30 04:47:51.916686 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:47:51.916674    5898 cpu_manager.go:414] "containerMap: removing container" podUID="10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5" containerName="console"
Oct 30 04:47:51.916729 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:47:51.916689    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5" containerName="console"
Oct 30 04:47:51.916770 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:47:51.916747    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="10b0b5a4-7f6b-4b56-b5c7-08713e4ba0d5" containerName="console"
Oct 30 04:47:51.919729 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:47:51.919707    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv8xdv"
Oct 30 04:47:51.948562 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:47:51.948536    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-lvzvx\""
Oct 30 04:47:51.948562 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:47:51.948547    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 04:47:51.948718 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:47:51.948589    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 04:47:51.949031 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:47:51.949014    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv8xdv"]
Oct 30 04:47:51.993514 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:47:51.993488    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/150266be-35ef-4e62-a690-4b295ed5d423-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv8xdv\" (UID: \"150266be-35ef-4e62-a690-4b295ed5d423\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv8xdv"
Oct 30 04:47:51.993630 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:47:51.993523    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/150266be-35ef-4e62-a690-4b295ed5d423-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv8xdv\" (UID: \"150266be-35ef-4e62-a690-4b295ed5d423\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv8xdv"
Oct 30 04:47:51.993630 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:47:51.993553    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb4bq\" (UniqueName: \"kubernetes.io/projected/150266be-35ef-4e62-a690-4b295ed5d423-kube-api-access-qb4bq\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv8xdv\" (UID: \"150266be-35ef-4e62-a690-4b295ed5d423\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv8xdv"
Oct 30 04:47:52.094223 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:47:52.094199    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb4bq\" (UniqueName: \"kubernetes.io/projected/150266be-35ef-4e62-a690-4b295ed5d423-kube-api-access-qb4bq\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv8xdv\" (UID: \"150266be-35ef-4e62-a690-4b295ed5d423\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv8xdv"
Oct 30 04:47:52.094611 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:47:52.094242    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/150266be-35ef-4e62-a690-4b295ed5d423-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv8xdv\" (UID: \"150266be-35ef-4e62-a690-4b295ed5d423\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv8xdv"
Oct 30 04:47:52.094611 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:47:52.094268    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/150266be-35ef-4e62-a690-4b295ed5d423-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv8xdv\" (UID: \"150266be-35ef-4e62-a690-4b295ed5d423\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv8xdv"
Oct 30 04:47:52.094611 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:47:52.094588    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/150266be-35ef-4e62-a690-4b295ed5d423-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv8xdv\" (UID: \"150266be-35ef-4e62-a690-4b295ed5d423\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv8xdv"
Oct 30 04:47:52.094713 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:47:52.094612    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/150266be-35ef-4e62-a690-4b295ed5d423-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv8xdv\" (UID: \"150266be-35ef-4e62-a690-4b295ed5d423\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv8xdv"
Oct 30 04:47:52.112810 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:47:52.112782    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb4bq\" (UniqueName: \"kubernetes.io/projected/150266be-35ef-4e62-a690-4b295ed5d423-kube-api-access-qb4bq\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv8xdv\" (UID: \"150266be-35ef-4e62-a690-4b295ed5d423\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv8xdv"
Oct 30 04:47:52.234102 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:47:52.234047    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv8xdv"
Oct 30 04:47:52.525406 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:47:52.525184    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv8xdv"]
Oct 30 04:47:52.529782 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:47:52.529748    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod150266be_35ef_4e62_a690_4b295ed5d423.slice/crio-fa32535bf69b97ccfeccc9776ae32dba191a9cc2bcddd428de3478eff131e4e4 WatchSource:0}: Error finding container fa32535bf69b97ccfeccc9776ae32dba191a9cc2bcddd428de3478eff131e4e4: Status 404 returned error can't find the container with id fa32535bf69b97ccfeccc9776ae32dba191a9cc2bcddd428de3478eff131e4e4
Oct 30 04:47:52.531812 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:47:52.531792    5898 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 04:47:52.585412 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:47:52.585385    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv8xdv" event={"ID":"150266be-35ef-4e62-a690-4b295ed5d423","Type":"ContainerStarted","Data":"fa32535bf69b97ccfeccc9776ae32dba191a9cc2bcddd428de3478eff131e4e4"}
Oct 30 04:47:56.819123 ip-10-0-135-98 kubenswrapper[5898]: E1030 04:47:56.819083    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 04:47:57.830429 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:47:57.830391    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:47:57.922602 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:47:57.922567    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:47:58.608280 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:47:58.608248    5898 generic.go:356] "Generic (PLEG): container finished" podID="150266be-35ef-4e62-a690-4b295ed5d423" containerID="4e3007b3055983bb299624ba7cb5046a196383d23035c5b75e07a6ad187e86a0" exitCode=0
Oct 30 04:47:58.608461 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:47:58.608314    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv8xdv" event={"ID":"150266be-35ef-4e62-a690-4b295ed5d423","Type":"ContainerDied","Data":"4e3007b3055983bb299624ba7cb5046a196383d23035c5b75e07a6ad187e86a0"}
Oct 30 04:48:00.616943 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:00.616884    5898 generic.go:356] "Generic (PLEG): container finished" podID="150266be-35ef-4e62-a690-4b295ed5d423" containerID="5cdd32a8a74522156567ae4ba11ef58e1d846185852229711876056f0a589b61" exitCode=0
Oct 30 04:48:00.617313 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:00.616971    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv8xdv" event={"ID":"150266be-35ef-4e62-a690-4b295ed5d423","Type":"ContainerDied","Data":"5cdd32a8a74522156567ae4ba11ef58e1d846185852229711876056f0a589b61"}
Oct 30 04:48:06.641436 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:06.641405    5898 generic.go:356] "Generic (PLEG): container finished" podID="150266be-35ef-4e62-a690-4b295ed5d423" containerID="c3e1da3b841ad487c25d25d9bda3abc2b5ddf2fc5ccaba2621b09f2b3a9cd574" exitCode=0
Oct 30 04:48:06.641790 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:06.641483    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv8xdv" event={"ID":"150266be-35ef-4e62-a690-4b295ed5d423","Type":"ContainerDied","Data":"c3e1da3b841ad487c25d25d9bda3abc2b5ddf2fc5ccaba2621b09f2b3a9cd574"}
Oct 30 04:48:07.764223 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:07.764196    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv8xdv"
Oct 30 04:48:07.924671 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:07.924589    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/150266be-35ef-4e62-a690-4b295ed5d423-bundle\" (UniqueName: \"kubernetes.io/empty-dir/150266be-35ef-4e62-a690-4b295ed5d423-bundle\") pod \"150266be-35ef-4e62-a690-4b295ed5d423\" (UID: \"150266be-35ef-4e62-a690-4b295ed5d423\") "
Oct 30 04:48:07.924671 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:07.924666    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/150266be-35ef-4e62-a690-4b295ed5d423-util\" (UniqueName: \"kubernetes.io/empty-dir/150266be-35ef-4e62-a690-4b295ed5d423-util\") pod \"150266be-35ef-4e62-a690-4b295ed5d423\" (UID: \"150266be-35ef-4e62-a690-4b295ed5d423\") "
Oct 30 04:48:07.924934 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:07.924699    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/150266be-35ef-4e62-a690-4b295ed5d423-kube-api-access-qb4bq\" (UniqueName: \"kubernetes.io/projected/150266be-35ef-4e62-a690-4b295ed5d423-kube-api-access-qb4bq\") pod \"150266be-35ef-4e62-a690-4b295ed5d423\" (UID: \"150266be-35ef-4e62-a690-4b295ed5d423\") "
Oct 30 04:48:07.925348 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:07.925320    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/150266be-35ef-4e62-a690-4b295ed5d423-bundle" pod "150266be-35ef-4e62-a690-4b295ed5d423" (UID: "150266be-35ef-4e62-a690-4b295ed5d423"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:48:07.926871 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:07.926843    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/150266be-35ef-4e62-a690-4b295ed5d423-kube-api-access-qb4bq" pod "150266be-35ef-4e62-a690-4b295ed5d423" (UID: "150266be-35ef-4e62-a690-4b295ed5d423"). InnerVolumeSpecName "kube-api-access-qb4bq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:48:07.929225 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:07.929205    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/150266be-35ef-4e62-a690-4b295ed5d423-util" pod "150266be-35ef-4e62-a690-4b295ed5d423" (UID: "150266be-35ef-4e62-a690-4b295ed5d423"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:48:08.025663 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:08.025637    5898 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/150266be-35ef-4e62-a690-4b295ed5d423-util\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 04:48:08.025663 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:08.025661    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb4bq\" (UniqueName: \"kubernetes.io/projected/150266be-35ef-4e62-a690-4b295ed5d423-kube-api-access-qb4bq\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 04:48:08.025807 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:08.025671    5898 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/150266be-35ef-4e62-a690-4b295ed5d423-bundle\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 04:48:08.650599 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:08.650560    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv8xdv" event={"ID":"150266be-35ef-4e62-a690-4b295ed5d423","Type":"ContainerDied","Data":"fa32535bf69b97ccfeccc9776ae32dba191a9cc2bcddd428de3478eff131e4e4"}
Oct 30 04:48:08.650599 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:08.650583    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv8xdv"
Oct 30 04:48:08.650793 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:08.650616    5898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa32535bf69b97ccfeccc9776ae32dba191a9cc2bcddd428de3478eff131e4e4"
Oct 30 04:48:12.922422 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:12.922389    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:48:14.025102 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:14.025070    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:48:14.816698 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:14.816661    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-tn79k"]
Oct 30 04:48:14.817031 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:14.817003    5898 cpu_manager.go:414] "containerMap: removing container" podUID="150266be-35ef-4e62-a690-4b295ed5d423" containerName="util"
Oct 30 04:48:14.817031 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:14.817032    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="150266be-35ef-4e62-a690-4b295ed5d423" containerName="util"
Oct 30 04:48:14.817187 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:14.817053    5898 cpu_manager.go:414] "containerMap: removing container" podUID="150266be-35ef-4e62-a690-4b295ed5d423" containerName="pull"
Oct 30 04:48:14.817187 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:14.817061    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="150266be-35ef-4e62-a690-4b295ed5d423" containerName="pull"
Oct 30 04:48:14.817187 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:14.817074    5898 cpu_manager.go:414] "containerMap: removing container" podUID="150266be-35ef-4e62-a690-4b295ed5d423" containerName="extract"
Oct 30 04:48:14.817187 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:14.817083    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="150266be-35ef-4e62-a690-4b295ed5d423" containerName="extract"
Oct 30 04:48:14.817187 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:14.817154    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="150266be-35ef-4e62-a690-4b295ed5d423" containerName="extract"
Oct 30 04:48:14.857023 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:14.856997    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-tn79k"]
Oct 30 04:48:14.857189 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:14.857112    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-tn79k"
Oct 30 04:48:14.870298 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:14.870274    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef86aebe-681b-4a8f-9369-dcaa4d7b02e2-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-tn79k\" (UID: \"ef86aebe-681b-4a8f-9369-dcaa4d7b02e2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-tn79k"
Oct 30 04:48:14.870399 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:14.870325    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs2jc\" (UniqueName: \"kubernetes.io/projected/ef86aebe-681b-4a8f-9369-dcaa4d7b02e2-kube-api-access-rs2jc\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-tn79k\" (UID: \"ef86aebe-681b-4a8f-9369-dcaa4d7b02e2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-tn79k"
Oct 30 04:48:14.871609 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:14.871592    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Oct 30 04:48:14.871697 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:14.871682    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Oct 30 04:48:14.874245 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:14.874229    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-dzcll\""
Oct 30 04:48:14.971046 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:14.971019    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef86aebe-681b-4a8f-9369-dcaa4d7b02e2-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-tn79k\" (UID: \"ef86aebe-681b-4a8f-9369-dcaa4d7b02e2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-tn79k"
Oct 30 04:48:14.971185 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:14.971056    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs2jc\" (UniqueName: \"kubernetes.io/projected/ef86aebe-681b-4a8f-9369-dcaa4d7b02e2-kube-api-access-rs2jc\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-tn79k\" (UID: \"ef86aebe-681b-4a8f-9369-dcaa4d7b02e2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-tn79k"
Oct 30 04:48:14.971357 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:14.971338    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef86aebe-681b-4a8f-9369-dcaa4d7b02e2-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-tn79k\" (UID: \"ef86aebe-681b-4a8f-9369-dcaa4d7b02e2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-tn79k"
Oct 30 04:48:14.989518 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:14.989499    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs2jc\" (UniqueName: \"kubernetes.io/projected/ef86aebe-681b-4a8f-9369-dcaa4d7b02e2-kube-api-access-rs2jc\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-tn79k\" (UID: \"ef86aebe-681b-4a8f-9369-dcaa4d7b02e2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-tn79k"
Oct 30 04:48:15.178965 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:15.178932    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-tn79k"
Oct 30 04:48:15.516056 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:15.515974    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-tn79k"]
Oct 30 04:48:15.519069 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:48:15.519037    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef86aebe_681b_4a8f_9369_dcaa4d7b02e2.slice/crio-47e1c7ca82b86797e1078342a52ef3f3aebce22b4612801612ae82d16a5de056 WatchSource:0}: Error finding container 47e1c7ca82b86797e1078342a52ef3f3aebce22b4612801612ae82d16a5de056: Status 404 returned error can't find the container with id 47e1c7ca82b86797e1078342a52ef3f3aebce22b4612801612ae82d16a5de056
Oct 30 04:48:15.674836 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:15.674804    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-tn79k" event={"ID":"ef86aebe-681b-4a8f-9369-dcaa4d7b02e2","Type":"ContainerStarted","Data":"47e1c7ca82b86797e1078342a52ef3f3aebce22b4612801612ae82d16a5de056"}
Oct 30 04:48:16.161286 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:16.161250    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:48:16.262141 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:16.262098    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:48:17.683914 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:17.683866    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-tn79k" event={"ID":"ef86aebe-681b-4a8f-9369-dcaa4d7b02e2","Type":"ContainerStarted","Data":"2a0b057865db8adcdb9b4a930c54f0e2edc2cc451bf643fd7de1714848aaf676"}
Oct 30 04:48:17.747309 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:17.747258    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-tn79k" podStartSLOduration=1.739002378 podStartE2EDuration="3.747245285s" totalImagesPullingTime="2.008242907s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:48:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:48:17.73513655 +0000 UTC m=+226.958956454" watchObservedRunningTime="2026-10-30 04:48:17.747245285 +0000 UTC m=+226.971065185"
Oct 30 04:48:18.298945 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:18.298875    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:48:18.418627 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:18.418592    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:48:18.630309 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:18.630275    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:48:18.732583 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:18.732532    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:48:18.936794 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:18.936713    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85pkgxx"]
Oct 30 04:48:18.958794 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:18.958767    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85pkgxx"]
Oct 30 04:48:18.958971 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:18.958866    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85pkgxx"
Oct 30 04:48:18.984198 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:18.984175    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 04:48:18.984300 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:18.984204    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"61b1ff68c7255ee5c5329fe19a5d96293a9578-ce91a85d-dockercfg-krgx2\""
Oct 30 04:48:18.984300 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:18.984235    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 04:48:18.998623 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:18.998599    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72jqm\" (UniqueName: \"kubernetes.io/projected/0f67f736-723c-4b75-9663-ecc5f82885a8-kube-api-access-72jqm\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85pkgxx\" (UID: \"0f67f736-723c-4b75-9663-ecc5f82885a8\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85pkgxx"
Oct 30 04:48:18.998717 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:18.998632    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0f67f736-723c-4b75-9663-ecc5f82885a8-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85pkgxx\" (UID: \"0f67f736-723c-4b75-9663-ecc5f82885a8\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85pkgxx"
Oct 30 04:48:18.998717 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:18.998669    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0f67f736-723c-4b75-9663-ecc5f82885a8-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85pkgxx\" (UID: \"0f67f736-723c-4b75-9663-ecc5f82885a8\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85pkgxx"
Oct 30 04:48:19.003408 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:19.003382    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-kz7km"]
Oct 30 04:48:19.017057 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:19.017038    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kz7km"
Oct 30 04:48:19.027397 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:19.027378    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-kz7km"]
Oct 30 04:48:19.073542 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:19.073515    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 30 04:48:19.073648 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:19.073596    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-jfrms\""
Oct 30 04:48:19.073765 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:19.073730    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 30 04:48:19.099606 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:19.099581    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6b75122d-eff3-404c-aaaa-524c014fa7fc-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-kz7km\" (UID: \"6b75122d-eff3-404c-aaaa-524c014fa7fc\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kz7km"
Oct 30 04:48:19.099721 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:19.099627    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r6cp\" (UniqueName: \"kubernetes.io/projected/6b75122d-eff3-404c-aaaa-524c014fa7fc-kube-api-access-9r6cp\") pod \"cert-manager-webhook-5dbbbf75cc-kz7km\" (UID: \"6b75122d-eff3-404c-aaaa-524c014fa7fc\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kz7km"
Oct 30 04:48:19.099721 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:19.099651    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0f67f736-723c-4b75-9663-ecc5f82885a8-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85pkgxx\" (UID: \"0f67f736-723c-4b75-9663-ecc5f82885a8\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85pkgxx"
Oct 30 04:48:19.099848 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:19.099819    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72jqm\" (UniqueName: \"kubernetes.io/projected/0f67f736-723c-4b75-9663-ecc5f82885a8-kube-api-access-72jqm\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85pkgxx\" (UID: \"0f67f736-723c-4b75-9663-ecc5f82885a8\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85pkgxx"
Oct 30 04:48:19.099940 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:19.099850    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0f67f736-723c-4b75-9663-ecc5f82885a8-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85pkgxx\" (UID: \"0f67f736-723c-4b75-9663-ecc5f82885a8\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85pkgxx"
Oct 30 04:48:19.100060 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:19.100042    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0f67f736-723c-4b75-9663-ecc5f82885a8-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85pkgxx\" (UID: \"0f67f736-723c-4b75-9663-ecc5f82885a8\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85pkgxx"
Oct 30 04:48:19.100132 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:19.100118    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0f67f736-723c-4b75-9663-ecc5f82885a8-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85pkgxx\" (UID: \"0f67f736-723c-4b75-9663-ecc5f82885a8\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85pkgxx"
Oct 30 04:48:19.125382 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:19.125353    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72jqm\" (UniqueName: \"kubernetes.io/projected/0f67f736-723c-4b75-9663-ecc5f82885a8-kube-api-access-72jqm\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85pkgxx\" (UID: \"0f67f736-723c-4b75-9663-ecc5f82885a8\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85pkgxx"
Oct 30 04:48:19.200216 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:19.200159    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6b75122d-eff3-404c-aaaa-524c014fa7fc-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-kz7km\" (UID: \"6b75122d-eff3-404c-aaaa-524c014fa7fc\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kz7km"
Oct 30 04:48:19.200216 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:19.200204    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r6cp\" (UniqueName: \"kubernetes.io/projected/6b75122d-eff3-404c-aaaa-524c014fa7fc-kube-api-access-9r6cp\") pod \"cert-manager-webhook-5dbbbf75cc-kz7km\" (UID: \"6b75122d-eff3-404c-aaaa-524c014fa7fc\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kz7km"
Oct 30 04:48:19.223103 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:19.223073    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6b75122d-eff3-404c-aaaa-524c014fa7fc-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-kz7km\" (UID: \"6b75122d-eff3-404c-aaaa-524c014fa7fc\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kz7km"
Oct 30 04:48:19.223191 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:19.223128    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r6cp\" (UniqueName: \"kubernetes.io/projected/6b75122d-eff3-404c-aaaa-524c014fa7fc-kube-api-access-9r6cp\") pod \"cert-manager-webhook-5dbbbf75cc-kz7km\" (UID: \"6b75122d-eff3-404c-aaaa-524c014fa7fc\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kz7km"
Oct 30 04:48:19.312057 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:19.312029    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85pkgxx"
Oct 30 04:48:19.359733 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:19.359706    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kz7km"
Oct 30 04:48:19.627053 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:19.627021    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85pkgxx"]
Oct 30 04:48:19.853868 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:48:19.630020    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f67f736_723c_4b75_9663_ecc5f82885a8.slice/crio-1f0fb79f34a5e5bf743bcffb7b7155b4ac88d2104d3a399bd726325889ba6bbe WatchSource:0}: Error finding container 1f0fb79f34a5e5bf743bcffb7b7155b4ac88d2104d3a399bd726325889ba6bbe: Status 404 returned error can't find the container with id 1f0fb79f34a5e5bf743bcffb7b7155b4ac88d2104d3a399bd726325889ba6bbe
Oct 30 04:48:19.853868 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:19.672700    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-kz7km"]
Oct 30 04:48:19.853868 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:48:19.676536    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b75122d_eff3_404c_aaaa_524c014fa7fc.slice/crio-664621df859ce80fb45594d9f35d8ce29874aae1a9b36eea928ded6cb9ea4f85 WatchSource:0}: Error finding container 664621df859ce80fb45594d9f35d8ce29874aae1a9b36eea928ded6cb9ea4f85: Status 404 returned error can't find the container with id 664621df859ce80fb45594d9f35d8ce29874aae1a9b36eea928ded6cb9ea4f85
Oct 30 04:48:19.853868 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:19.692096    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85pkgxx" event={"ID":"0f67f736-723c-4b75-9663-ecc5f82885a8","Type":"ContainerStarted","Data":"1f0fb79f34a5e5bf743bcffb7b7155b4ac88d2104d3a399bd726325889ba6bbe"}
Oct 30 04:48:19.853868 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:19.693210    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kz7km" event={"ID":"6b75122d-eff3-404c-aaaa-524c014fa7fc","Type":"ContainerStarted","Data":"664621df859ce80fb45594d9f35d8ce29874aae1a9b36eea928ded6cb9ea4f85"}
Oct 30 04:48:20.699171 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:20.699130    5898 generic.go:356] "Generic (PLEG): container finished" podID="0f67f736-723c-4b75-9663-ecc5f82885a8" containerID="191a21218a9b4f8fa5d79491d8e15c34eb89d8a68a97aca421d1b63445c466b4" exitCode=0
Oct 30 04:48:20.699361 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:20.699215    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85pkgxx" event={"ID":"0f67f736-723c-4b75-9663-ecc5f82885a8","Type":"ContainerDied","Data":"191a21218a9b4f8fa5d79491d8e15c34eb89d8a68a97aca421d1b63445c466b4"}
Oct 30 04:48:21.079979 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:21.079912    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:48:21.270914 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:21.268971    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-hw7dh"]
Oct 30 04:48:21.271741 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:21.271706    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hw7dh"
Oct 30 04:48:21.285315 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:21.285247    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:48:21.288876 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:21.288838    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-79v9n\""
Oct 30 04:48:21.289534 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:21.289358    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-hw7dh"]
Oct 30 04:48:21.321366 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:21.321331    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbzf2\" (UniqueName: \"kubernetes.io/projected/0d9a7b2b-2e4b-46f6-86cc-e3f41a76ed1d-kube-api-access-kbzf2\") pod \"cert-manager-cainjector-5c57bbd4fc-hw7dh\" (UID: \"0d9a7b2b-2e4b-46f6-86cc-e3f41a76ed1d\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hw7dh"
Oct 30 04:48:21.321491 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:21.321409    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0d9a7b2b-2e4b-46f6-86cc-e3f41a76ed1d-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-hw7dh\" (UID: \"0d9a7b2b-2e4b-46f6-86cc-e3f41a76ed1d\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hw7dh"
Oct 30 04:48:21.423019 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:21.422390    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0d9a7b2b-2e4b-46f6-86cc-e3f41a76ed1d-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-hw7dh\" (UID: \"0d9a7b2b-2e4b-46f6-86cc-e3f41a76ed1d\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hw7dh"
Oct 30 04:48:21.423019 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:21.422459    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbzf2\" (UniqueName: \"kubernetes.io/projected/0d9a7b2b-2e4b-46f6-86cc-e3f41a76ed1d-kube-api-access-kbzf2\") pod \"cert-manager-cainjector-5c57bbd4fc-hw7dh\" (UID: \"0d9a7b2b-2e4b-46f6-86cc-e3f41a76ed1d\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hw7dh"
Oct 30 04:48:21.441156 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:21.441126    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0d9a7b2b-2e4b-46f6-86cc-e3f41a76ed1d-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-hw7dh\" (UID: \"0d9a7b2b-2e4b-46f6-86cc-e3f41a76ed1d\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hw7dh"
Oct 30 04:48:21.442594 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:21.442571    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbzf2\" (UniqueName: \"kubernetes.io/projected/0d9a7b2b-2e4b-46f6-86cc-e3f41a76ed1d-kube-api-access-kbzf2\") pod \"cert-manager-cainjector-5c57bbd4fc-hw7dh\" (UID: \"0d9a7b2b-2e4b-46f6-86cc-e3f41a76ed1d\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hw7dh"
Oct 30 04:48:21.584704 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:21.584670    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hw7dh"
Oct 30 04:48:21.888327 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:21.888294    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-hw7dh"]
Oct 30 04:48:21.892351 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:48:21.892318    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d9a7b2b_2e4b_46f6_86cc_e3f41a76ed1d.slice/crio-bc9c5dd5dfae33df57d8607c7d800b208e4647ef1fefe5d857eded045480f4ce WatchSource:0}: Error finding container bc9c5dd5dfae33df57d8607c7d800b208e4647ef1fefe5d857eded045480f4ce: Status 404 returned error can't find the container with id bc9c5dd5dfae33df57d8607c7d800b208e4647ef1fefe5d857eded045480f4ce
Oct 30 04:48:22.710392 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:22.710357    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kz7km" event={"ID":"6b75122d-eff3-404c-aaaa-524c014fa7fc","Type":"ContainerStarted","Data":"95fe5ea075177a3903a318f44d1992a7b656b3411cd17bd82e166265fe0afc45"}
Oct 30 04:48:22.710799 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:22.710545    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kz7km"
Oct 30 04:48:22.720373 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:22.720315    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85pkgxx" event={"ID":"0f67f736-723c-4b75-9663-ecc5f82885a8","Type":"ContainerStarted","Data":"eae50c094602812535e2e6bac177ff29343f2d9f9a1b781a02cf742ece7a35e5"}
Oct 30 04:48:22.722391 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:22.722359    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hw7dh" event={"ID":"0d9a7b2b-2e4b-46f6-86cc-e3f41a76ed1d","Type":"ContainerStarted","Data":"aeda8316ca2997ce4b8b27f6ad070e8e6f6812427779e354a3c1b52008290514"}
Oct 30 04:48:22.722499 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:22.722403    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hw7dh" event={"ID":"0d9a7b2b-2e4b-46f6-86cc-e3f41a76ed1d","Type":"ContainerStarted","Data":"bc9c5dd5dfae33df57d8607c7d800b208e4647ef1fefe5d857eded045480f4ce"}
Oct 30 04:48:22.747686 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:22.747640    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kz7km" podStartSLOduration=2.086147109 podStartE2EDuration="4.747628491s" totalImagesPullingTime="2.661481382s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:48:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:48:22.746519261 +0000 UTC m=+231.970339163" watchObservedRunningTime="2026-10-30 04:48:22.747628491 +0000 UTC m=+231.971448391"
Oct 30 04:48:23.114466 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:23.114436    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:48:23.256620 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:23.256590    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:48:23.728265 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:23.728236    5898 generic.go:356] "Generic (PLEG): container finished" podID="0f67f736-723c-4b75-9663-ecc5f82885a8" containerID="eae50c094602812535e2e6bac177ff29343f2d9f9a1b781a02cf742ece7a35e5" exitCode=0
Oct 30 04:48:23.728684 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:23.728328    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85pkgxx" event={"ID":"0f67f736-723c-4b75-9663-ecc5f82885a8","Type":"ContainerDied","Data":"eae50c094602812535e2e6bac177ff29343f2d9f9a1b781a02cf742ece7a35e5"}
Oct 30 04:48:23.766013 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:23.765895    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hw7dh" podStartSLOduration=2.281576534 podStartE2EDuration="2.765876458s" totalImagesPullingTime="484.299924ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:48:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:48:22.817954563 +0000 UTC m=+232.041774484" watchObservedRunningTime="2026-10-30 04:48:23.765876458 +0000 UTC m=+232.989696360"
Oct 30 04:48:24.733796 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:24.733759    5898 generic.go:356] "Generic (PLEG): container finished" podID="0f67f736-723c-4b75-9663-ecc5f82885a8" containerID="3b0d03e25e59eef4998df638dfd6448db9e56a5a9edf34db142aaeccbd3f0711" exitCode=0
Oct 30 04:48:24.734211 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:24.733843    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85pkgxx" event={"ID":"0f67f736-723c-4b75-9663-ecc5f82885a8","Type":"ContainerDied","Data":"3b0d03e25e59eef4998df638dfd6448db9e56a5a9edf34db142aaeccbd3f0711"}
Oct 30 04:48:25.871617 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:25.871594    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85pkgxx"
Oct 30 04:48:25.960322 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:25.960289    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f67f736-723c-4b75-9663-ecc5f82885a8-util\" (UniqueName: \"kubernetes.io/empty-dir/0f67f736-723c-4b75-9663-ecc5f82885a8-util\") pod \"0f67f736-723c-4b75-9663-ecc5f82885a8\" (UID: \"0f67f736-723c-4b75-9663-ecc5f82885a8\") "
Oct 30 04:48:25.960477 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:25.960341    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f67f736-723c-4b75-9663-ecc5f82885a8-bundle\" (UniqueName: \"kubernetes.io/empty-dir/0f67f736-723c-4b75-9663-ecc5f82885a8-bundle\") pod \"0f67f736-723c-4b75-9663-ecc5f82885a8\" (UID: \"0f67f736-723c-4b75-9663-ecc5f82885a8\") "
Oct 30 04:48:25.960477 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:25.960393    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f67f736-723c-4b75-9663-ecc5f82885a8-kube-api-access-72jqm\" (UniqueName: \"kubernetes.io/projected/0f67f736-723c-4b75-9663-ecc5f82885a8-kube-api-access-72jqm\") pod \"0f67f736-723c-4b75-9663-ecc5f82885a8\" (UID: \"0f67f736-723c-4b75-9663-ecc5f82885a8\") "
Oct 30 04:48:25.960782 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:25.960738    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f67f736-723c-4b75-9663-ecc5f82885a8-bundle" pod "0f67f736-723c-4b75-9663-ecc5f82885a8" (UID: "0f67f736-723c-4b75-9663-ecc5f82885a8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:48:25.960969 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:25.960863    5898 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0f67f736-723c-4b75-9663-ecc5f82885a8-bundle\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 04:48:25.962558 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:25.962534    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f67f736-723c-4b75-9663-ecc5f82885a8-kube-api-access-72jqm" pod "0f67f736-723c-4b75-9663-ecc5f82885a8" (UID: "0f67f736-723c-4b75-9663-ecc5f82885a8"). InnerVolumeSpecName "kube-api-access-72jqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:48:25.965210 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:25.965176    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f67f736-723c-4b75-9663-ecc5f82885a8-util" pod "0f67f736-723c-4b75-9663-ecc5f82885a8" (UID: "0f67f736-723c-4b75-9663-ecc5f82885a8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:48:26.061225 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:26.061156    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72jqm\" (UniqueName: \"kubernetes.io/projected/0f67f736-723c-4b75-9663-ecc5f82885a8-kube-api-access-72jqm\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 04:48:26.061225 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:26.061181    5898 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0f67f736-723c-4b75-9663-ecc5f82885a8-util\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 04:48:26.743451 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:26.743416    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85pkgxx" event={"ID":"0f67f736-723c-4b75-9663-ecc5f82885a8","Type":"ContainerDied","Data":"1f0fb79f34a5e5bf743bcffb7b7155b4ac88d2104d3a399bd726325889ba6bbe"}
Oct 30 04:48:26.743451 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:26.743455    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85pkgxx"
Oct 30 04:48:26.743623 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:26.743464    5898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f0fb79f34a5e5bf743bcffb7b7155b4ac88d2104d3a399bd726325889ba6bbe"
Oct 30 04:48:26.817051 ip-10-0-135-98 kubenswrapper[5898]: E1030 04:48:26.817016    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 04:48:28.730586 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:28.730558    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kz7km"
Oct 30 04:48:29.073303 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:29.073274    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:48:31.228826 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:31.228791    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:48:31.323869 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:31.323813    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:48:33.024699 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:33.024660    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-jr66t"]
Oct 30 04:48:33.025086 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:33.024930    5898 cpu_manager.go:414] "containerMap: removing container" podUID="0f67f736-723c-4b75-9663-ecc5f82885a8" containerName="pull"
Oct 30 04:48:33.025086 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:33.024941    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f67f736-723c-4b75-9663-ecc5f82885a8" containerName="pull"
Oct 30 04:48:33.025086 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:33.024950    5898 cpu_manager.go:414] "containerMap: removing container" podUID="0f67f736-723c-4b75-9663-ecc5f82885a8" containerName="extract"
Oct 30 04:48:33.025086 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:33.024955    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f67f736-723c-4b75-9663-ecc5f82885a8" containerName="extract"
Oct 30 04:48:33.025086 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:33.024967    5898 cpu_manager.go:414] "containerMap: removing container" podUID="0f67f736-723c-4b75-9663-ecc5f82885a8" containerName="util"
Oct 30 04:48:33.025086 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:33.024972    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f67f736-723c-4b75-9663-ecc5f82885a8" containerName="util"
Oct 30 04:48:33.025086 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:33.025030    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f67f736-723c-4b75-9663-ecc5f82885a8" containerName="extract"
Oct 30 04:48:33.028721 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:33.028700    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-8465994787-jr66t"
Oct 30 04:48:33.049702 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:33.049678    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-p2w2b\""
Oct 30 04:48:33.050392 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:33.050372    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-jr66t"]
Oct 30 04:48:33.054789 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:33.054768    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Oct 30 04:48:33.054789 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:33.054781    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Oct 30 04:48:33.111051 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:33.111019    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2pdc\" (UniqueName: \"kubernetes.io/projected/78c7bdd6-6958-445a-9a98-637875cac2ba-kube-api-access-n2pdc\") pod \"openshift-lws-operator-8465994787-jr66t\" (UID: \"78c7bdd6-6958-445a-9a98-637875cac2ba\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-jr66t"
Oct 30 04:48:33.111180 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:33.111055    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78c7bdd6-6958-445a-9a98-637875cac2ba-tmp\") pod \"openshift-lws-operator-8465994787-jr66t\" (UID: \"78c7bdd6-6958-445a-9a98-637875cac2ba\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-jr66t"
Oct 30 04:48:33.212422 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:33.212391    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2pdc\" (UniqueName: \"kubernetes.io/projected/78c7bdd6-6958-445a-9a98-637875cac2ba-kube-api-access-n2pdc\") pod \"openshift-lws-operator-8465994787-jr66t\" (UID: \"78c7bdd6-6958-445a-9a98-637875cac2ba\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-jr66t"
Oct 30 04:48:33.212422 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:33.212426    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78c7bdd6-6958-445a-9a98-637875cac2ba-tmp\") pod \"openshift-lws-operator-8465994787-jr66t\" (UID: \"78c7bdd6-6958-445a-9a98-637875cac2ba\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-jr66t"
Oct 30 04:48:33.212765 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:33.212749    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78c7bdd6-6958-445a-9a98-637875cac2ba-tmp\") pod \"openshift-lws-operator-8465994787-jr66t\" (UID: \"78c7bdd6-6958-445a-9a98-637875cac2ba\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-jr66t"
Oct 30 04:48:33.234349 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:33.234322    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2pdc\" (UniqueName: \"kubernetes.io/projected/78c7bdd6-6958-445a-9a98-637875cac2ba-kube-api-access-n2pdc\") pod \"openshift-lws-operator-8465994787-jr66t\" (UID: \"78c7bdd6-6958-445a-9a98-637875cac2ba\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-jr66t"
Oct 30 04:48:33.340175 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:33.340139    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-8465994787-jr66t"
Oct 30 04:48:33.628647 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:33.628582    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-jr66t"]
Oct 30 04:48:33.631635 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:48:33.631604    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod78c7bdd6_6958_445a_9a98_637875cac2ba.slice/crio-8d3e539763adc6b23f09a9b1c96e49b7248ab8151c682aad12a67f9f73679a88 WatchSource:0}: Error finding container 8d3e539763adc6b23f09a9b1c96e49b7248ab8151c682aad12a67f9f73679a88: Status 404 returned error can't find the container with id 8d3e539763adc6b23f09a9b1c96e49b7248ab8151c682aad12a67f9f73679a88
Oct 30 04:48:33.768973 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:33.768886    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-8465994787-jr66t" event={"ID":"78c7bdd6-6958-445a-9a98-637875cac2ba","Type":"ContainerStarted","Data":"8d3e539763adc6b23f09a9b1c96e49b7248ab8151c682aad12a67f9f73679a88"}
Oct 30 04:48:35.778767 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:35.778684    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-8465994787-jr66t" event={"ID":"78c7bdd6-6958-445a-9a98-637875cac2ba","Type":"ContainerStarted","Data":"14f806b5b765e8f24243d4d0e8c1307bf5aca6f8bf0f72d13f61ac282185335b"}
Oct 30 04:48:35.816277 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:35.816220    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-8465994787-jr66t" podStartSLOduration=1.091616262 podStartE2EDuration="2.816202309s" totalImagesPullingTime="1.724586047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:48:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:48:35.815093422 +0000 UTC m=+245.038913323" watchObservedRunningTime="2026-10-30 04:48:35.816202309 +0000 UTC m=+245.040022213"
Oct 30 04:48:36.908298 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:36.908269    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:48:37.007683 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:37.007649    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:48:38.754022 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:38.753989    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bslsf"]
Oct 30 04:48:38.757442 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:38.757426    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bslsf"
Oct 30 04:48:38.784037 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:38.784011    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 04:48:38.784037 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:38.784011    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 04:48:38.784297 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:38.784047    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-h6snz\""
Oct 30 04:48:38.784843 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:38.784819    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bslsf"]
Oct 30 04:48:38.852725 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:38.852697    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/80361ceb-fc03-4102-ad15-4d15ea624041-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bslsf\" (UID: \"80361ceb-fc03-4102-ad15-4d15ea624041\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bslsf"
Oct 30 04:48:38.852871 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:38.852742    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/80361ceb-fc03-4102-ad15-4d15ea624041-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bslsf\" (UID: \"80361ceb-fc03-4102-ad15-4d15ea624041\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bslsf"
Oct 30 04:48:38.852871 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:38.852815    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlxpl\" (UniqueName: \"kubernetes.io/projected/80361ceb-fc03-4102-ad15-4d15ea624041-kube-api-access-dlxpl\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bslsf\" (UID: \"80361ceb-fc03-4102-ad15-4d15ea624041\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bslsf"
Oct 30 04:48:38.900019 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:38.899979    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:48:38.954201 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:38.954169    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/80361ceb-fc03-4102-ad15-4d15ea624041-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bslsf\" (UID: \"80361ceb-fc03-4102-ad15-4d15ea624041\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bslsf"
Oct 30 04:48:38.954339 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:38.954209    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlxpl\" (UniqueName: \"kubernetes.io/projected/80361ceb-fc03-4102-ad15-4d15ea624041-kube-api-access-dlxpl\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bslsf\" (UID: \"80361ceb-fc03-4102-ad15-4d15ea624041\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bslsf"
Oct 30 04:48:38.954339 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:38.954262    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/80361ceb-fc03-4102-ad15-4d15ea624041-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bslsf\" (UID: \"80361ceb-fc03-4102-ad15-4d15ea624041\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bslsf"
Oct 30 04:48:38.954672 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:38.954644    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/80361ceb-fc03-4102-ad15-4d15ea624041-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bslsf\" (UID: \"80361ceb-fc03-4102-ad15-4d15ea624041\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bslsf"
Oct 30 04:48:38.954779 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:38.954747    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/80361ceb-fc03-4102-ad15-4d15ea624041-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bslsf\" (UID: \"80361ceb-fc03-4102-ad15-4d15ea624041\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bslsf"
Oct 30 04:48:38.973215 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:38.973192    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlxpl\" (UniqueName: \"kubernetes.io/projected/80361ceb-fc03-4102-ad15-4d15ea624041-kube-api-access-dlxpl\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bslsf\" (UID: \"80361ceb-fc03-4102-ad15-4d15ea624041\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bslsf"
Oct 30 04:48:39.073065 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:39.073046    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bslsf"
Oct 30 04:48:39.363421 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:39.363351    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bslsf"]
Oct 30 04:48:39.366796 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:48:39.366767    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80361ceb_fc03_4102_ad15_4d15ea624041.slice/crio-f7d5f37c8c89e8d4ba47bdb44f339798211db3568cf57a93b12ccc2d6d33bf70 WatchSource:0}: Error finding container f7d5f37c8c89e8d4ba47bdb44f339798211db3568cf57a93b12ccc2d6d33bf70: Status 404 returned error can't find the container with id f7d5f37c8c89e8d4ba47bdb44f339798211db3568cf57a93b12ccc2d6d33bf70
Oct 30 04:48:39.794742 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:39.794660    5898 generic.go:356] "Generic (PLEG): container finished" podID="80361ceb-fc03-4102-ad15-4d15ea624041" containerID="3c4e550cd036c7adb61f920388391a2e0a001d4e6893d84fc9dc0a5aa4743fb5" exitCode=0
Oct 30 04:48:39.794742 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:39.794712    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bslsf" event={"ID":"80361ceb-fc03-4102-ad15-4d15ea624041","Type":"ContainerDied","Data":"3c4e550cd036c7adb61f920388391a2e0a001d4e6893d84fc9dc0a5aa4743fb5"}
Oct 30 04:48:39.795143 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:39.794746    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bslsf" event={"ID":"80361ceb-fc03-4102-ad15-4d15ea624041","Type":"ContainerStarted","Data":"f7d5f37c8c89e8d4ba47bdb44f339798211db3568cf57a93b12ccc2d6d33bf70"}
Oct 30 04:48:40.800250 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:40.800220    5898 generic.go:356] "Generic (PLEG): container finished" podID="80361ceb-fc03-4102-ad15-4d15ea624041" containerID="0fe60246a968e25c5eb77a2db1dbdd5850c7e94c138df8e2f8c565afc6cb6494" exitCode=0
Oct 30 04:48:40.800633 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:40.800292    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bslsf" event={"ID":"80361ceb-fc03-4102-ad15-4d15ea624041","Type":"ContainerDied","Data":"0fe60246a968e25c5eb77a2db1dbdd5850c7e94c138df8e2f8c565afc6cb6494"}
Oct 30 04:48:41.806863 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:41.806826    5898 generic.go:356] "Generic (PLEG): container finished" podID="80361ceb-fc03-4102-ad15-4d15ea624041" containerID="2a840b1b7b6ef51642efebf23eb105596ec1ebecc5d4fa12d8585f4e6ba592f4" exitCode=0
Oct 30 04:48:41.807253 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:41.806891    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bslsf" event={"ID":"80361ceb-fc03-4102-ad15-4d15ea624041","Type":"ContainerDied","Data":"2a840b1b7b6ef51642efebf23eb105596ec1ebecc5d4fa12d8585f4e6ba592f4"}
Oct 30 04:48:42.932491 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:42.932468    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bslsf"
Oct 30 04:48:42.980840 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:42.980816    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80361ceb-fc03-4102-ad15-4d15ea624041-bundle\" (UniqueName: \"kubernetes.io/empty-dir/80361ceb-fc03-4102-ad15-4d15ea624041-bundle\") pod \"80361ceb-fc03-4102-ad15-4d15ea624041\" (UID: \"80361ceb-fc03-4102-ad15-4d15ea624041\") "
Oct 30 04:48:42.980986 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:42.980869    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80361ceb-fc03-4102-ad15-4d15ea624041-util\" (UniqueName: \"kubernetes.io/empty-dir/80361ceb-fc03-4102-ad15-4d15ea624041-util\") pod \"80361ceb-fc03-4102-ad15-4d15ea624041\" (UID: \"80361ceb-fc03-4102-ad15-4d15ea624041\") "
Oct 30 04:48:42.980986 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:42.980915    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80361ceb-fc03-4102-ad15-4d15ea624041-kube-api-access-dlxpl\" (UniqueName: \"kubernetes.io/projected/80361ceb-fc03-4102-ad15-4d15ea624041-kube-api-access-dlxpl\") pod \"80361ceb-fc03-4102-ad15-4d15ea624041\" (UID: \"80361ceb-fc03-4102-ad15-4d15ea624041\") "
Oct 30 04:48:42.981556 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:42.981523    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80361ceb-fc03-4102-ad15-4d15ea624041-bundle" pod "80361ceb-fc03-4102-ad15-4d15ea624041" (UID: "80361ceb-fc03-4102-ad15-4d15ea624041"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:48:42.983193 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:42.983162    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80361ceb-fc03-4102-ad15-4d15ea624041-kube-api-access-dlxpl" pod "80361ceb-fc03-4102-ad15-4d15ea624041" (UID: "80361ceb-fc03-4102-ad15-4d15ea624041"). InnerVolumeSpecName "kube-api-access-dlxpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:48:42.989279 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:42.989241    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80361ceb-fc03-4102-ad15-4d15ea624041-util" pod "80361ceb-fc03-4102-ad15-4d15ea624041" (UID: "80361ceb-fc03-4102-ad15-4d15ea624041"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:48:43.082190 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:43.082164    5898 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/80361ceb-fc03-4102-ad15-4d15ea624041-bundle\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 04:48:43.082190 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:43.082189    5898 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/80361ceb-fc03-4102-ad15-4d15ea624041-util\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 04:48:43.082341 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:43.082204    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlxpl\" (UniqueName: \"kubernetes.io/projected/80361ceb-fc03-4102-ad15-4d15ea624041-kube-api-access-dlxpl\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 04:48:43.816635 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:43.816597    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bslsf" event={"ID":"80361ceb-fc03-4102-ad15-4d15ea624041","Type":"ContainerDied","Data":"f7d5f37c8c89e8d4ba47bdb44f339798211db3568cf57a93b12ccc2d6d33bf70"}
Oct 30 04:48:43.816844 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:43.816648    5898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7d5f37c8c89e8d4ba47bdb44f339798211db3568cf57a93b12ccc2d6d33bf70"
Oct 30 04:48:43.816844 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:43.816652    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bslsf"
Oct 30 04:48:46.471389 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:46.471353    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:48:46.551268 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:46.551211    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:48:52.349738 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:52.349704    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:48:52.503816 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:52.503777    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:48:52.890792 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:52.890754    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:48:53.052386 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:53.052320    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8484dd48c9-sxchv"]
Oct 30 04:48:53.052586 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:53.052574    5898 cpu_manager.go:414] "containerMap: removing container" podUID="80361ceb-fc03-4102-ad15-4d15ea624041" containerName="util"
Oct 30 04:48:53.052621 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:53.052587    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80361ceb-fc03-4102-ad15-4d15ea624041" containerName="util"
Oct 30 04:48:53.052621 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:53.052599    5898 cpu_manager.go:414] "containerMap: removing container" podUID="80361ceb-fc03-4102-ad15-4d15ea624041" containerName="pull"
Oct 30 04:48:53.052621 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:53.052603    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80361ceb-fc03-4102-ad15-4d15ea624041" containerName="pull"
Oct 30 04:48:53.052713 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:53.052622    5898 cpu_manager.go:414] "containerMap: removing container" podUID="80361ceb-fc03-4102-ad15-4d15ea624041" containerName="extract"
Oct 30 04:48:53.052713 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:53.052627    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80361ceb-fc03-4102-ad15-4d15ea624041" containerName="extract"
Oct 30 04:48:53.052713 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:53.052674    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="80361ceb-fc03-4102-ad15-4d15ea624041" containerName="extract"
Oct 30 04:48:53.097142 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:53.097114    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8484dd48c9-sxchv"]
Oct 30 04:48:53.097284 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:53.097219    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-sxchv"
Oct 30 04:48:53.155489 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:53.155457    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b85ad37a-53e3-4c9f-abab-7231172266fb-tmp\") pod \"opendatahub-operator-controller-manager-8484dd48c9-sxchv\" (UID: \"b85ad37a-53e3-4c9f-abab-7231172266fb\") " pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-sxchv"
Oct 30 04:48:53.155636 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:53.155499    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hndp6\" (UniqueName: \"kubernetes.io/projected/b85ad37a-53e3-4c9f-abab-7231172266fb-kube-api-access-hndp6\") pod \"opendatahub-operator-controller-manager-8484dd48c9-sxchv\" (UID: \"b85ad37a-53e3-4c9f-abab-7231172266fb\") " pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-sxchv"
Oct 30 04:48:53.155636 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:53.155526    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b85ad37a-53e3-4c9f-abab-7231172266fb-metrics-cert\") pod \"opendatahub-operator-controller-manager-8484dd48c9-sxchv\" (UID: \"b85ad37a-53e3-4c9f-abab-7231172266fb\") " pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-sxchv"
Oct 30 04:48:53.155711 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:53.155646    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b85ad37a-53e3-4c9f-abab-7231172266fb-webhook-cert\") pod \"opendatahub-operator-controller-manager-8484dd48c9-sxchv\" (UID: \"b85ad37a-53e3-4c9f-abab-7231172266fb\") " pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-sxchv"
Oct 30 04:48:53.155711 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:53.155691    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b85ad37a-53e3-4c9f-abab-7231172266fb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8484dd48c9-sxchv\" (UID: \"b85ad37a-53e3-4c9f-abab-7231172266fb\") " pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-sxchv"
Oct 30 04:48:53.155769 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:53.155739    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b85ad37a-53e3-4c9f-abab-7231172266fb-manifests\") pod \"opendatahub-operator-controller-manager-8484dd48c9-sxchv\" (UID: \"b85ad37a-53e3-4c9f-abab-7231172266fb\") " pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-sxchv"
Oct 30 04:48:53.182125 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:53.182095    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:48:53.206593 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:53.206571    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 04:48:53.207015 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:53.206992    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 04:48:53.207015 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:53.207007    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 04:48:53.207158 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:53.207036    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 04:48:53.207158 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:53.207106    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\""
Oct 30 04:48:53.207158 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:53.207155    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-wr5jk\""
Oct 30 04:48:53.256847 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:53.256821    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b85ad37a-53e3-4c9f-abab-7231172266fb-metrics-cert\") pod \"opendatahub-operator-controller-manager-8484dd48c9-sxchv\" (UID: \"b85ad37a-53e3-4c9f-abab-7231172266fb\") " pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-sxchv"
Oct 30 04:48:53.256951 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:53.256860    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b85ad37a-53e3-4c9f-abab-7231172266fb-webhook-cert\") pod \"opendatahub-operator-controller-manager-8484dd48c9-sxchv\" (UID: \"b85ad37a-53e3-4c9f-abab-7231172266fb\") " pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-sxchv"
Oct 30 04:48:53.256951 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:53.256920    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b85ad37a-53e3-4c9f-abab-7231172266fb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8484dd48c9-sxchv\" (UID: \"b85ad37a-53e3-4c9f-abab-7231172266fb\") " pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-sxchv"
Oct 30 04:48:53.257027 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:53.256949    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b85ad37a-53e3-4c9f-abab-7231172266fb-manifests\") pod \"opendatahub-operator-controller-manager-8484dd48c9-sxchv\" (UID: \"b85ad37a-53e3-4c9f-abab-7231172266fb\") " pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-sxchv"
Oct 30 04:48:53.257027 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:53.256980    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b85ad37a-53e3-4c9f-abab-7231172266fb-tmp\") pod \"opendatahub-operator-controller-manager-8484dd48c9-sxchv\" (UID: \"b85ad37a-53e3-4c9f-abab-7231172266fb\") " pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-sxchv"
Oct 30 04:48:53.257125 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:53.257025    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hndp6\" (UniqueName: \"kubernetes.io/projected/b85ad37a-53e3-4c9f-abab-7231172266fb-kube-api-access-hndp6\") pod \"opendatahub-operator-controller-manager-8484dd48c9-sxchv\" (UID: \"b85ad37a-53e3-4c9f-abab-7231172266fb\") " pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-sxchv"
Oct 30 04:48:53.257313 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:53.257290    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b85ad37a-53e3-4c9f-abab-7231172266fb-manifests\") pod \"opendatahub-operator-controller-manager-8484dd48c9-sxchv\" (UID: \"b85ad37a-53e3-4c9f-abab-7231172266fb\") " pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-sxchv"
Oct 30 04:48:53.259386 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:53.259360    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b85ad37a-53e3-4c9f-abab-7231172266fb-tmp\") pod \"opendatahub-operator-controller-manager-8484dd48c9-sxchv\" (UID: \"b85ad37a-53e3-4c9f-abab-7231172266fb\") " pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-sxchv"
Oct 30 04:48:53.259468 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:53.259443    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b85ad37a-53e3-4c9f-abab-7231172266fb-webhook-cert\") pod \"opendatahub-operator-controller-manager-8484dd48c9-sxchv\" (UID: \"b85ad37a-53e3-4c9f-abab-7231172266fb\") " pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-sxchv"
Oct 30 04:48:53.259508 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:53.259441    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b85ad37a-53e3-4c9f-abab-7231172266fb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8484dd48c9-sxchv\" (UID: \"b85ad37a-53e3-4c9f-abab-7231172266fb\") " pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-sxchv"
Oct 30 04:48:53.259624 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:53.259606    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b85ad37a-53e3-4c9f-abab-7231172266fb-metrics-cert\") pod \"opendatahub-operator-controller-manager-8484dd48c9-sxchv\" (UID: \"b85ad37a-53e3-4c9f-abab-7231172266fb\") " pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-sxchv"
Oct 30 04:48:53.275580 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:53.275556    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hndp6\" (UniqueName: \"kubernetes.io/projected/b85ad37a-53e3-4c9f-abab-7231172266fb-kube-api-access-hndp6\") pod \"opendatahub-operator-controller-manager-8484dd48c9-sxchv\" (UID: \"b85ad37a-53e3-4c9f-abab-7231172266fb\") " pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-sxchv"
Oct 30 04:48:53.280140 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:53.280117    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:48:53.381072 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:53.381043    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:48:53.441205 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:53.441182    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-sxchv"
Oct 30 04:48:53.481836 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:53.481811    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:48:53.823467 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:53.823439    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8484dd48c9-sxchv"]
Oct 30 04:48:53.826556 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:48:53.826525    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb85ad37a_53e3_4c9f_abab_7231172266fb.slice/crio-19e03db36c2b62a9e82de25f164944d1361bf370f136d81c5a9ba7131d144284 WatchSource:0}: Error finding container 19e03db36c2b62a9e82de25f164944d1361bf370f136d81c5a9ba7131d144284: Status 404 returned error can't find the container with id 19e03db36c2b62a9e82de25f164944d1361bf370f136d81c5a9ba7131d144284
Oct 30 04:48:53.859755 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:53.859723    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-sxchv" event={"ID":"b85ad37a-53e3-4c9f-abab-7231172266fb","Type":"ContainerStarted","Data":"19e03db36c2b62a9e82de25f164944d1361bf370f136d81c5a9ba7131d144284"}
Oct 30 04:48:53.861134 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:53.861111    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:48:56.816521 ip-10-0-135-98 kubenswrapper[5898]: E1030 04:48:56.816438    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 04:48:56.873887 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:56.873858    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-sxchv" event={"ID":"b85ad37a-53e3-4c9f-abab-7231172266fb","Type":"ContainerStarted","Data":"fc838af1516c3b3a3147bbc96212dcdf2cf4473f824ef76e7bd0fa31b8b86f8f"}
Oct 30 04:48:57.878969 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:57.878932    5898 generic.go:356] "Generic (PLEG): container finished" podID="b85ad37a-53e3-4c9f-abab-7231172266fb" containerID="fc838af1516c3b3a3147bbc96212dcdf2cf4473f824ef76e7bd0fa31b8b86f8f" exitCode=0
Oct 30 04:48:57.879344 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:57.879018    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-sxchv" event={"ID":"b85ad37a-53e3-4c9f-abab-7231172266fb","Type":"ContainerDied","Data":"fc838af1516c3b3a3147bbc96212dcdf2cf4473f824ef76e7bd0fa31b8b86f8f"}
Oct 30 04:48:58.887556 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:58.887526    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-sxchv"
Oct 30 04:48:58.887969 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:58.887576    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-sxchv" event={"ID":"b85ad37a-53e3-4c9f-abab-7231172266fb","Type":"ContainerStarted","Data":"5921c9a51534abac013f10130f7442da4b036adbbc6ce9617a479c6ded43ebd0"}
Oct 30 04:48:58.957111 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:48:58.957065    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-sxchv" podStartSLOduration=2.5544794 podStartE2EDuration="5.957049862s" totalImagesPullingTime="2.968766634s" totalInitContainerRuntime="433.803828ms" isStatefulPod=false podCreationTimestamp="2026-10-30 04:48:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:48:58.944656754 +0000 UTC m=+268.168476654" watchObservedRunningTime="2026-10-30 04:48:58.957049862 +0000 UTC m=+268.180869769"
Oct 30 04:49:05.328087 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:05.328048    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:49:05.428835 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:05.428802    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:49:05.528798 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:05.528770    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:49:05.948860 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:05.948825    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:49:06.027057 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:06.027016    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:49:06.459761 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:06.459731    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:49:06.541806 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:06.541777    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:49:09.891382 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:09.891356    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-8484dd48c9-sxchv"
Oct 30 04:49:11.767098 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:11.767062    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7f587b55d5-cv8nm"]
Oct 30 04:49:11.774820 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:11.774798    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7f587b55d5-cv8nm"
Oct 30 04:49:11.785952 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:11.785930    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7f587b55d5-cv8nm"]
Oct 30 04:49:11.874322 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:11.874287    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Oct 30 04:49:11.874322 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:11.874311    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Oct 30 04:49:11.874322 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:11.874322    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-qck5r\""
Oct 30 04:49:11.874536 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:11.874371    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Oct 30 04:49:11.905196 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:11.905164    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a0f33b2-1008-4c4a-9d2a-20a41a7834ae-cert\") pod \"lws-controller-manager-7f587b55d5-cv8nm\" (UID: \"6a0f33b2-1008-4c4a-9d2a-20a41a7834ae\") " pod="openshift-lws-operator/lws-controller-manager-7f587b55d5-cv8nm"
Oct 30 04:49:11.905315 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:11.905205    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6a0f33b2-1008-4c4a-9d2a-20a41a7834ae-metrics-cert\") pod \"lws-controller-manager-7f587b55d5-cv8nm\" (UID: \"6a0f33b2-1008-4c4a-9d2a-20a41a7834ae\") " pod="openshift-lws-operator/lws-controller-manager-7f587b55d5-cv8nm"
Oct 30 04:49:11.905315 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:11.905226    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6a0f33b2-1008-4c4a-9d2a-20a41a7834ae-manager-config\") pod \"lws-controller-manager-7f587b55d5-cv8nm\" (UID: \"6a0f33b2-1008-4c4a-9d2a-20a41a7834ae\") " pod="openshift-lws-operator/lws-controller-manager-7f587b55d5-cv8nm"
Oct 30 04:49:11.905396 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:11.905374    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drdvj\" (UniqueName: \"kubernetes.io/projected/6a0f33b2-1008-4c4a-9d2a-20a41a7834ae-kube-api-access-drdvj\") pod \"lws-controller-manager-7f587b55d5-cv8nm\" (UID: \"6a0f33b2-1008-4c4a-9d2a-20a41a7834ae\") " pod="openshift-lws-operator/lws-controller-manager-7f587b55d5-cv8nm"
Oct 30 04:49:12.006208 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:12.006165    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drdvj\" (UniqueName: \"kubernetes.io/projected/6a0f33b2-1008-4c4a-9d2a-20a41a7834ae-kube-api-access-drdvj\") pod \"lws-controller-manager-7f587b55d5-cv8nm\" (UID: \"6a0f33b2-1008-4c4a-9d2a-20a41a7834ae\") " pod="openshift-lws-operator/lws-controller-manager-7f587b55d5-cv8nm"
Oct 30 04:49:12.006375 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:12.006228    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a0f33b2-1008-4c4a-9d2a-20a41a7834ae-cert\") pod \"lws-controller-manager-7f587b55d5-cv8nm\" (UID: \"6a0f33b2-1008-4c4a-9d2a-20a41a7834ae\") " pod="openshift-lws-operator/lws-controller-manager-7f587b55d5-cv8nm"
Oct 30 04:49:12.006375 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:12.006261    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6a0f33b2-1008-4c4a-9d2a-20a41a7834ae-metrics-cert\") pod \"lws-controller-manager-7f587b55d5-cv8nm\" (UID: \"6a0f33b2-1008-4c4a-9d2a-20a41a7834ae\") " pod="openshift-lws-operator/lws-controller-manager-7f587b55d5-cv8nm"
Oct 30 04:49:12.006444 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:12.006415    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6a0f33b2-1008-4c4a-9d2a-20a41a7834ae-manager-config\") pod \"lws-controller-manager-7f587b55d5-cv8nm\" (UID: \"6a0f33b2-1008-4c4a-9d2a-20a41a7834ae\") " pod="openshift-lws-operator/lws-controller-manager-7f587b55d5-cv8nm"
Oct 30 04:49:12.007106 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:12.007080    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6a0f33b2-1008-4c4a-9d2a-20a41a7834ae-manager-config\") pod \"lws-controller-manager-7f587b55d5-cv8nm\" (UID: \"6a0f33b2-1008-4c4a-9d2a-20a41a7834ae\") " pod="openshift-lws-operator/lws-controller-manager-7f587b55d5-cv8nm"
Oct 30 04:49:12.008720 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:12.008695    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a0f33b2-1008-4c4a-9d2a-20a41a7834ae-cert\") pod \"lws-controller-manager-7f587b55d5-cv8nm\" (UID: \"6a0f33b2-1008-4c4a-9d2a-20a41a7834ae\") " pod="openshift-lws-operator/lws-controller-manager-7f587b55d5-cv8nm"
Oct 30 04:49:12.008811 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:12.008757    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6a0f33b2-1008-4c4a-9d2a-20a41a7834ae-metrics-cert\") pod \"lws-controller-manager-7f587b55d5-cv8nm\" (UID: \"6a0f33b2-1008-4c4a-9d2a-20a41a7834ae\") " pod="openshift-lws-operator/lws-controller-manager-7f587b55d5-cv8nm"
Oct 30 04:49:12.031143 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:12.031074    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drdvj\" (UniqueName: \"kubernetes.io/projected/6a0f33b2-1008-4c4a-9d2a-20a41a7834ae-kube-api-access-drdvj\") pod \"lws-controller-manager-7f587b55d5-cv8nm\" (UID: \"6a0f33b2-1008-4c4a-9d2a-20a41a7834ae\") " pod="openshift-lws-operator/lws-controller-manager-7f587b55d5-cv8nm"
Oct 30 04:49:12.086532 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:12.086503    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7f587b55d5-cv8nm"
Oct 30 04:49:12.269184 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:12.269149    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:49:12.363655 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:12.363622    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:49:12.385623 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:12.385600    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7f587b55d5-cv8nm"]
Oct 30 04:49:12.388565 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:49:12.388539    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a0f33b2_1008_4c4a_9d2a_20a41a7834ae.slice/crio-8309e7a96adff5feb31ffeddead48e5c2462c9b6c2d0ea2ac9d72fe647f00404 WatchSource:0}: Error finding container 8309e7a96adff5feb31ffeddead48e5c2462c9b6c2d0ea2ac9d72fe647f00404: Status 404 returned error can't find the container with id 8309e7a96adff5feb31ffeddead48e5c2462c9b6c2d0ea2ac9d72fe647f00404
Oct 30 04:49:12.938378 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:12.938344    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7f587b55d5-cv8nm" event={"ID":"6a0f33b2-1008-4c4a-9d2a-20a41a7834ae","Type":"ContainerStarted","Data":"8309e7a96adff5feb31ffeddead48e5c2462c9b6c2d0ea2ac9d72fe647f00404"}
Oct 30 04:49:14.950728 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:14.950690    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7f587b55d5-cv8nm" event={"ID":"6a0f33b2-1008-4c4a-9d2a-20a41a7834ae","Type":"ContainerStarted","Data":"011a7bedcbf12209380f66177bc3cd2430976c5bb45685aa95c42575a708359e"}
Oct 30 04:49:14.951172 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:14.950845    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7f587b55d5-cv8nm"
Oct 30 04:49:14.986786 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:14.986731    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7f587b55d5-cv8nm" podStartSLOduration=2.34087564 podStartE2EDuration="3.986718007s" totalImagesPullingTime="1.645842367s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:49:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:49:14.986212467 +0000 UTC m=+284.210032368" watchObservedRunningTime="2026-10-30 04:49:14.986718007 +0000 UTC m=+284.210537928"
Oct 30 04:49:15.508925 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:15.508872    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:49:15.831366 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:15.831335    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:49:15.943401 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:15.943372    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:49:16.058044 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:16.058011    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:49:16.281004 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:16.280889    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:49:16.997797 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:16.997766    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:49:17.117585 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:17.117554    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:49:17.904453 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:17.904418    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:49:21.095704 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:21.095672    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:49:21.173764 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:21.173737    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:49:21.682968 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:21.682934    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-nwq4j"]
Oct 30 04:49:21.686757 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:21.686731    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-nwq4j"
Oct 30 04:49:21.700817 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:21.700793    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 04:49:21.700944 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:21.700887    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 04:49:21.701169 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:21.701153    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 04:49:21.713446 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:21.713415    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-nwq4j"]
Oct 30 04:49:21.719786 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:21.719767    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-sqtnn\""
Oct 30 04:49:21.736690 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:21.736662    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:49:21.777885 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:21.777852    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5db038b-f1f8-434b-bd6d-48707dce8b5c-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-nwq4j\" (UID: \"d5db038b-f1f8-434b-bd6d-48707dce8b5c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-nwq4j"
Oct 30 04:49:21.778016 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:21.777920    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5db038b-f1f8-434b-bd6d-48707dce8b5c-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-nwq4j\" (UID: \"d5db038b-f1f8-434b-bd6d-48707dce8b5c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-nwq4j"
Oct 30 04:49:21.778016 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:21.777984    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5db038b-f1f8-434b-bd6d-48707dce8b5c-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-nwq4j\" (UID: \"d5db038b-f1f8-434b-bd6d-48707dce8b5c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-nwq4j"
Oct 30 04:49:21.778016 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:21.778011    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5db038b-f1f8-434b-bd6d-48707dce8b5c-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-nwq4j\" (UID: \"d5db038b-f1f8-434b-bd6d-48707dce8b5c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-nwq4j"
Oct 30 04:49:21.778115 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:21.778032    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbfl8\" (UniqueName: \"kubernetes.io/projected/d5db038b-f1f8-434b-bd6d-48707dce8b5c-kube-api-access-nbfl8\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-nwq4j\" (UID: \"d5db038b-f1f8-434b-bd6d-48707dce8b5c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-nwq4j"
Oct 30 04:49:21.778115 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:21.778053    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5db038b-f1f8-434b-bd6d-48707dce8b5c-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-nwq4j\" (UID: \"d5db038b-f1f8-434b-bd6d-48707dce8b5c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-nwq4j"
Oct 30 04:49:21.778193 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:21.778172    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5db038b-f1f8-434b-bd6d-48707dce8b5c-istio-token\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-nwq4j\" (UID: \"d5db038b-f1f8-434b-bd6d-48707dce8b5c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-nwq4j"
Oct 30 04:49:21.778228 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:21.778214    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5db038b-f1f8-434b-bd6d-48707dce8b5c-istio-data\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-nwq4j\" (UID: \"d5db038b-f1f8-434b-bd6d-48707dce8b5c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-nwq4j"
Oct 30 04:49:21.778260 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:21.778242    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5db038b-f1f8-434b-bd6d-48707dce8b5c-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-nwq4j\" (UID: \"d5db038b-f1f8-434b-bd6d-48707dce8b5c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-nwq4j"
Oct 30 04:49:21.878829 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:21.878802    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5db038b-f1f8-434b-bd6d-48707dce8b5c-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-nwq4j\" (UID: \"d5db038b-f1f8-434b-bd6d-48707dce8b5c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-nwq4j"
Oct 30 04:49:21.878983 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:21.878837    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5db038b-f1f8-434b-bd6d-48707dce8b5c-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-nwq4j\" (UID: \"d5db038b-f1f8-434b-bd6d-48707dce8b5c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-nwq4j"
Oct 30 04:49:21.878983 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:21.878858    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbfl8\" (UniqueName: \"kubernetes.io/projected/d5db038b-f1f8-434b-bd6d-48707dce8b5c-kube-api-access-nbfl8\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-nwq4j\" (UID: \"d5db038b-f1f8-434b-bd6d-48707dce8b5c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-nwq4j"
Oct 30 04:49:21.878983 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:21.878880    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5db038b-f1f8-434b-bd6d-48707dce8b5c-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-nwq4j\" (UID: \"d5db038b-f1f8-434b-bd6d-48707dce8b5c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-nwq4j"
Oct 30 04:49:21.878983 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:21.878923    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5db038b-f1f8-434b-bd6d-48707dce8b5c-istio-token\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-nwq4j\" (UID: \"d5db038b-f1f8-434b-bd6d-48707dce8b5c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-nwq4j"
Oct 30 04:49:21.878983 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:21.878951    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5db038b-f1f8-434b-bd6d-48707dce8b5c-istio-data\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-nwq4j\" (UID: \"d5db038b-f1f8-434b-bd6d-48707dce8b5c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-nwq4j"
Oct 30 04:49:21.878983 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:21.878981    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5db038b-f1f8-434b-bd6d-48707dce8b5c-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-nwq4j\" (UID: \"d5db038b-f1f8-434b-bd6d-48707dce8b5c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-nwq4j"
Oct 30 04:49:21.879281 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:21.879006    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5db038b-f1f8-434b-bd6d-48707dce8b5c-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-nwq4j\" (UID: \"d5db038b-f1f8-434b-bd6d-48707dce8b5c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-nwq4j"
Oct 30 04:49:21.879281 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:21.879052    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5db038b-f1f8-434b-bd6d-48707dce8b5c-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-nwq4j\" (UID: \"d5db038b-f1f8-434b-bd6d-48707dce8b5c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-nwq4j"
Oct 30 04:49:21.879435 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:21.879410    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5db038b-f1f8-434b-bd6d-48707dce8b5c-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-nwq4j\" (UID: \"d5db038b-f1f8-434b-bd6d-48707dce8b5c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-nwq4j"
Oct 30 04:49:21.879497 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:21.879417    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5db038b-f1f8-434b-bd6d-48707dce8b5c-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-nwq4j\" (UID: \"d5db038b-f1f8-434b-bd6d-48707dce8b5c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-nwq4j"
Oct 30 04:49:21.879497 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:21.879485    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5db038b-f1f8-434b-bd6d-48707dce8b5c-istio-data\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-nwq4j\" (UID: \"d5db038b-f1f8-434b-bd6d-48707dce8b5c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-nwq4j"
Oct 30 04:49:21.879603 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:21.879518    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5db038b-f1f8-434b-bd6d-48707dce8b5c-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-nwq4j\" (UID: \"d5db038b-f1f8-434b-bd6d-48707dce8b5c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-nwq4j"
Oct 30 04:49:21.879664 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:21.879642    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5db038b-f1f8-434b-bd6d-48707dce8b5c-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-nwq4j\" (UID: \"d5db038b-f1f8-434b-bd6d-48707dce8b5c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-nwq4j"
Oct 30 04:49:21.881277 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:21.881255    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5db038b-f1f8-434b-bd6d-48707dce8b5c-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-nwq4j\" (UID: \"d5db038b-f1f8-434b-bd6d-48707dce8b5c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-nwq4j"
Oct 30 04:49:21.881385 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:21.881366    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5db038b-f1f8-434b-bd6d-48707dce8b5c-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-nwq4j\" (UID: \"d5db038b-f1f8-434b-bd6d-48707dce8b5c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-nwq4j"
Oct 30 04:49:21.891866 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:21.891845    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:49:21.896779 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:21.896754    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5db038b-f1f8-434b-bd6d-48707dce8b5c-istio-token\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-nwq4j\" (UID: \"d5db038b-f1f8-434b-bd6d-48707dce8b5c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-nwq4j"
Oct 30 04:49:21.896949 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:21.896929    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbfl8\" (UniqueName: \"kubernetes.io/projected/d5db038b-f1f8-434b-bd6d-48707dce8b5c-kube-api-access-nbfl8\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-nwq4j\" (UID: \"d5db038b-f1f8-434b-bd6d-48707dce8b5c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-nwq4j"
Oct 30 04:49:22.002443 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:22.002380    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-nwq4j"
Oct 30 04:49:22.320828 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:22.320801    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-nwq4j"]
Oct 30 04:49:22.323546 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:49:22.323515    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5db038b_f1f8_434b_bd6d_48707dce8b5c.slice/crio-d2c5bdbb584e44d401d422745237569bf8748f0bf344d30a81b6674f8f7d8c84 WatchSource:0}: Error finding container d2c5bdbb584e44d401d422745237569bf8748f0bf344d30a81b6674f8f7d8c84: Status 404 returned error can't find the container with id d2c5bdbb584e44d401d422745237569bf8748f0bf344d30a81b6674f8f7d8c84
Oct 30 04:49:22.393167 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:22.393138    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:49:22.984210 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:22.984168    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-nwq4j" event={"ID":"d5db038b-f1f8-434b-bd6d-48707dce8b5c","Type":"ContainerStarted","Data":"d2c5bdbb584e44d401d422745237569bf8748f0bf344d30a81b6674f8f7d8c84"}
Oct 30 04:49:24.761438 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:24.761403    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 04:49:24.761722 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:24.761475    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 04:49:24.761722 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:24.761504    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 04:49:24.994032 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:24.994002    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-nwq4j" event={"ID":"d5db038b-f1f8-434b-bd6d-48707dce8b5c","Type":"ContainerStarted","Data":"4f04481c451ae625958f3d48556a412c8d9638941afca8d21b4b1398c6449483"}
Oct 30 04:49:25.003425 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:25.003407    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-nwq4j"
Oct 30 04:49:25.004792 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:25.004761    5898 patch_prober.go:29] interesting pod/data-science-gateway-data-science-gateway-class-67df64d57-nwq4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" start-of-body=
Oct 30 04:49:25.004930 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:25.004806    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-nwq4j" podUID="d5db038b-f1f8-434b-bd6d-48707dce8b5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused"
Oct 30 04:49:25.046040 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:25.045959    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-nwq4j" podStartSLOduration=1.610702635 podStartE2EDuration="4.045947197s" totalImagesPullingTime="2.435244562s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:49:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:49:25.045455853 +0000 UTC m=+294.269275773" watchObservedRunningTime="2026-10-30 04:49:25.045947197 +0000 UTC m=+294.269767092"
Oct 30 04:49:25.956783 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:25.956754    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7f587b55d5-cv8nm"
Oct 30 04:49:26.006593 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:26.006563    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-nwq4j"
Oct 30 04:49:26.030777 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:26.030744    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:49:26.131246 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:26.131215    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:49:26.816893 ip-10-0-135-98 kubenswrapper[5898]: E1030 04:49:26.816855    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 04:49:27.002551 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:27.002523    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-nwq4j"
Oct 30 04:49:27.003502 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:27.003484    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-nwq4j"
Oct 30 04:49:28.703821 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:28.703777    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:49:28.779289 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:28.779259    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-c28c8"]
Oct 30 04:49:28.785745 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:28.785721    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-c28c8"
Oct 30 04:49:28.812804 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:28.812779    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-f9qt4\""
Oct 30 04:49:28.814098 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:28.814074    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-c28c8"]
Oct 30 04:49:28.836456 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:28.836431    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Oct 30 04:49:28.882163 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:28.882139    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:49:28.939306 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:28.939279    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td624\" (UniqueName: \"kubernetes.io/projected/25690daf-01ba-4364-8d5b-10712cf9d193-kube-api-access-td624\") pod \"kserve-module-controller-manager-5cfb77bf8c-c28c8\" (UID: \"25690daf-01ba-4364-8d5b-10712cf9d193\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-c28c8"
Oct 30 04:49:28.939443 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:28.939320    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/25690daf-01ba-4364-8d5b-10712cf9d193-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-c28c8\" (UID: \"25690daf-01ba-4364-8d5b-10712cf9d193\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-c28c8"
Oct 30 04:49:28.939443 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:28.939342    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/25690daf-01ba-4364-8d5b-10712cf9d193-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-c28c8\" (UID: \"25690daf-01ba-4364-8d5b-10712cf9d193\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-c28c8"
Oct 30 04:49:28.939443 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:28.939423    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/25690daf-01ba-4364-8d5b-10712cf9d193-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-c28c8\" (UID: \"25690daf-01ba-4364-8d5b-10712cf9d193\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-c28c8"
Oct 30 04:49:29.040617 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:29.040553    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td624\" (UniqueName: \"kubernetes.io/projected/25690daf-01ba-4364-8d5b-10712cf9d193-kube-api-access-td624\") pod \"kserve-module-controller-manager-5cfb77bf8c-c28c8\" (UID: \"25690daf-01ba-4364-8d5b-10712cf9d193\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-c28c8"
Oct 30 04:49:29.040617 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:29.040592    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/25690daf-01ba-4364-8d5b-10712cf9d193-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-c28c8\" (UID: \"25690daf-01ba-4364-8d5b-10712cf9d193\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-c28c8"
Oct 30 04:49:29.040617 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:29.040612    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/25690daf-01ba-4364-8d5b-10712cf9d193-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-c28c8\" (UID: \"25690daf-01ba-4364-8d5b-10712cf9d193\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-c28c8"
Oct 30 04:49:29.040830 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:29.040629    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/25690daf-01ba-4364-8d5b-10712cf9d193-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-c28c8\" (UID: \"25690daf-01ba-4364-8d5b-10712cf9d193\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-c28c8"
Oct 30 04:49:29.041001 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:29.040980    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/25690daf-01ba-4364-8d5b-10712cf9d193-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-c28c8\" (UID: \"25690daf-01ba-4364-8d5b-10712cf9d193\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-c28c8"
Oct 30 04:49:29.041091 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:29.041074    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/25690daf-01ba-4364-8d5b-10712cf9d193-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-c28c8\" (UID: \"25690daf-01ba-4364-8d5b-10712cf9d193\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-c28c8"
Oct 30 04:49:29.041476 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:29.041460    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/25690daf-01ba-4364-8d5b-10712cf9d193-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-c28c8\" (UID: \"25690daf-01ba-4364-8d5b-10712cf9d193\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-c28c8"
Oct 30 04:49:29.067171 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:29.067149    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-td624\" (UniqueName: \"kubernetes.io/projected/25690daf-01ba-4364-8d5b-10712cf9d193-kube-api-access-td624\") pod \"kserve-module-controller-manager-5cfb77bf8c-c28c8\" (UID: \"25690daf-01ba-4364-8d5b-10712cf9d193\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-c28c8"
Oct 30 04:49:29.097979 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:29.097958    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-c28c8"
Oct 30 04:49:29.402504 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:29.402474    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-c28c8"]
Oct 30 04:49:29.405552 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:49:29.405523    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25690daf_01ba_4364_8d5b_10712cf9d193.slice/crio-7821fbf6de981e2628bbd54e305ab37fb0c0f0357c8741d420f3d358318be832 WatchSource:0}: Error finding container 7821fbf6de981e2628bbd54e305ab37fb0c0f0357c8741d420f3d358318be832: Status 404 returned error can't find the container with id 7821fbf6de981e2628bbd54e305ab37fb0c0f0357c8741d420f3d358318be832
Oct 30 04:49:30.014434 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:30.014395    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-c28c8" event={"ID":"25690daf-01ba-4364-8d5b-10712cf9d193","Type":"ContainerStarted","Data":"7821fbf6de981e2628bbd54e305ab37fb0c0f0357c8741d420f3d358318be832"}
Oct 30 04:49:32.024374 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:32.024330    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-c28c8" event={"ID":"25690daf-01ba-4364-8d5b-10712cf9d193","Type":"ContainerStarted","Data":"f927e67b522f40b5ef2ce7e719741f6d7922ab09c77eef04b5f41ad9ba075ad4"}
Oct 30 04:49:32.024768 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:32.024485    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-c28c8"
Oct 30 04:49:32.062658 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:32.062609    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-c28c8" podStartSLOduration=1.581060091 podStartE2EDuration="4.062593989s" totalImagesPullingTime="2.481533898s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:49:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:49:32.061064074 +0000 UTC m=+301.284883975" watchObservedRunningTime="2026-10-30 04:49:32.062593989 +0000 UTC m=+301.286413891"
Oct 30 04:49:37.081817 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:37.081768    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:49:38.061127 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:38.061078    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:49:53.029789 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:49:53.029711    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-c28c8"
Oct 30 04:49:56.816764 ip-10-0-135-98 kubenswrapper[5898]: E1030 04:49:56.816724    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 04:50:06.706645 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:06.706613    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:50:06.783781 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:06.783713    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:50:09.668649 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:09.668573    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:50:09.758589 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:09.758553    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:50:15.365106 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:15.365066    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:50:15.465069 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:15.465034    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:50:15.564456 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:15.564418    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:50:15.664473 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:15.664382    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:50:15.675182 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:15.675156    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-4jgtx"]
Oct 30 04:50:15.680509 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:15.680486    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-4jgtx"
Oct 30 04:50:15.694024 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:15.693999    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-4jgtx"]
Oct 30 04:50:15.707076 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:15.707054    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Oct 30 04:50:15.707350 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:15.707332    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-pfh8z\""
Oct 30 04:50:15.708292 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:15.708275    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-57mpg"]
Oct 30 04:50:15.711481 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:15.711465    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-57mpg"
Oct 30 04:50:15.725935 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:15.725911    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-57mpg"]
Oct 30 04:50:15.756571 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:15.756546    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-t2c9k"]
Oct 30 04:50:15.760040 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:15.760020    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-t2c9k"
Oct 30 04:50:15.767091 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:15.767071    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:50:15.776695 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:15.776671    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-t2c9k"]
Oct 30 04:50:15.796844 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:15.796817    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0a25543c-936a-4521-b874-2ae3f9dcc61c-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-57mpg\" (UID: \"0a25543c-936a-4521-b874-2ae3f9dcc61c\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-57mpg"
Oct 30 04:50:15.796975 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:15.796873    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk4jx\" (UniqueName: \"kubernetes.io/projected/c712ba13-198b-48ba-9464-8b561f465646-kube-api-access-tk4jx\") pod \"kserve-controller-manager-776cc66d47-4jgtx\" (UID: \"c712ba13-198b-48ba-9464-8b561f465646\") " pod="opendatahub/kserve-controller-manager-776cc66d47-4jgtx"
Oct 30 04:50:15.797018 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:15.796977    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq7lx\" (UniqueName: \"kubernetes.io/projected/6a337a14-d243-45dc-8131-0d00b38564e8-kube-api-access-gq7lx\") pod \"model-serving-api-75bd89fdfc-t2c9k\" (UID: \"6a337a14-d243-45dc-8131-0d00b38564e8\") " pod="opendatahub/model-serving-api-75bd89fdfc-t2c9k"
Oct 30 04:50:15.797018 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:15.797000    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a337a14-d243-45dc-8131-0d00b38564e8-tls-certs\") pod \"model-serving-api-75bd89fdfc-t2c9k\" (UID: \"6a337a14-d243-45dc-8131-0d00b38564e8\") " pod="opendatahub/model-serving-api-75bd89fdfc-t2c9k"
Oct 30 04:50:15.797018 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:15.797015    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj825\" (UniqueName: \"kubernetes.io/projected/0a25543c-936a-4521-b874-2ae3f9dcc61c-kube-api-access-dj825\") pod \"llmisvc-controller-manager-869d5dcc5d-57mpg\" (UID: \"0a25543c-936a-4521-b874-2ae3f9dcc61c\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-57mpg"
Oct 30 04:50:15.797125 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:15.797106    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c712ba13-198b-48ba-9464-8b561f465646-cert\") pod \"kserve-controller-manager-776cc66d47-4jgtx\" (UID: \"c712ba13-198b-48ba-9464-8b561f465646\") " pod="opendatahub/kserve-controller-manager-776cc66d47-4jgtx"
Oct 30 04:50:15.799125 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:15.799110    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Oct 30 04:50:15.799198 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:15.799130    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-tdw4v\""
Oct 30 04:50:15.799198 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:15.799146    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Oct 30 04:50:15.799198 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:15.799176    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-99ld9\""
Oct 30 04:50:15.897800 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:15.897767    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c712ba13-198b-48ba-9464-8b561f465646-cert\") pod \"kserve-controller-manager-776cc66d47-4jgtx\" (UID: \"c712ba13-198b-48ba-9464-8b561f465646\") " pod="opendatahub/kserve-controller-manager-776cc66d47-4jgtx"
Oct 30 04:50:15.897979 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:15.897806    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0a25543c-936a-4521-b874-2ae3f9dcc61c-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-57mpg\" (UID: \"0a25543c-936a-4521-b874-2ae3f9dcc61c\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-57mpg"
Oct 30 04:50:15.897979 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:15.897840    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk4jx\" (UniqueName: \"kubernetes.io/projected/c712ba13-198b-48ba-9464-8b561f465646-kube-api-access-tk4jx\") pod \"kserve-controller-manager-776cc66d47-4jgtx\" (UID: \"c712ba13-198b-48ba-9464-8b561f465646\") " pod="opendatahub/kserve-controller-manager-776cc66d47-4jgtx"
Oct 30 04:50:15.897979 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:15.897874    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq7lx\" (UniqueName: \"kubernetes.io/projected/6a337a14-d243-45dc-8131-0d00b38564e8-kube-api-access-gq7lx\") pod \"model-serving-api-75bd89fdfc-t2c9k\" (UID: \"6a337a14-d243-45dc-8131-0d00b38564e8\") " pod="opendatahub/model-serving-api-75bd89fdfc-t2c9k"
Oct 30 04:50:15.897979 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:15.897891    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a337a14-d243-45dc-8131-0d00b38564e8-tls-certs\") pod \"model-serving-api-75bd89fdfc-t2c9k\" (UID: \"6a337a14-d243-45dc-8131-0d00b38564e8\") " pod="opendatahub/model-serving-api-75bd89fdfc-t2c9k"
Oct 30 04:50:15.897979 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:15.897922    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj825\" (UniqueName: \"kubernetes.io/projected/0a25543c-936a-4521-b874-2ae3f9dcc61c-kube-api-access-dj825\") pod \"llmisvc-controller-manager-869d5dcc5d-57mpg\" (UID: \"0a25543c-936a-4521-b874-2ae3f9dcc61c\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-57mpg"
Oct 30 04:50:15.900362 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:15.900337    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c712ba13-198b-48ba-9464-8b561f465646-cert\") pod \"kserve-controller-manager-776cc66d47-4jgtx\" (UID: \"c712ba13-198b-48ba-9464-8b561f465646\") " pod="opendatahub/kserve-controller-manager-776cc66d47-4jgtx"
Oct 30 04:50:15.900458 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:15.900396    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0a25543c-936a-4521-b874-2ae3f9dcc61c-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-57mpg\" (UID: \"0a25543c-936a-4521-b874-2ae3f9dcc61c\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-57mpg"
Oct 30 04:50:15.900458 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:15.900423    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a337a14-d243-45dc-8131-0d00b38564e8-tls-certs\") pod \"model-serving-api-75bd89fdfc-t2c9k\" (UID: \"6a337a14-d243-45dc-8131-0d00b38564e8\") " pod="opendatahub/model-serving-api-75bd89fdfc-t2c9k"
Oct 30 04:50:15.915945 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:15.915884    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq7lx\" (UniqueName: \"kubernetes.io/projected/6a337a14-d243-45dc-8131-0d00b38564e8-kube-api-access-gq7lx\") pod \"model-serving-api-75bd89fdfc-t2c9k\" (UID: \"6a337a14-d243-45dc-8131-0d00b38564e8\") " pod="opendatahub/model-serving-api-75bd89fdfc-t2c9k"
Oct 30 04:50:15.916093 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:15.916075    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk4jx\" (UniqueName: \"kubernetes.io/projected/c712ba13-198b-48ba-9464-8b561f465646-kube-api-access-tk4jx\") pod \"kserve-controller-manager-776cc66d47-4jgtx\" (UID: \"c712ba13-198b-48ba-9464-8b561f465646\") " pod="opendatahub/kserve-controller-manager-776cc66d47-4jgtx"
Oct 30 04:50:15.916345 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:15.916329    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj825\" (UniqueName: \"kubernetes.io/projected/0a25543c-936a-4521-b874-2ae3f9dcc61c-kube-api-access-dj825\") pod \"llmisvc-controller-manager-869d5dcc5d-57mpg\" (UID: \"0a25543c-936a-4521-b874-2ae3f9dcc61c\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-57mpg"
Oct 30 04:50:15.993135 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:15.993106    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-4jgtx"
Oct 30 04:50:16.023627 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:16.023603    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-57mpg"
Oct 30 04:50:16.071973 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:16.071941    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-t2c9k"
Oct 30 04:50:16.148850 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:16.148809    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:50:16.286008 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:16.285955    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-4jgtx"]
Oct 30 04:50:16.288590 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:50:16.288564    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc712ba13_198b_48ba_9464_8b561f465646.slice/crio-e1eda833f5258a050f99ca9e9714747b882f2eb0918911ede0180da7facc0d42 WatchSource:0}: Error finding container e1eda833f5258a050f99ca9e9714747b882f2eb0918911ede0180da7facc0d42: Status 404 returned error can't find the container with id e1eda833f5258a050f99ca9e9714747b882f2eb0918911ede0180da7facc0d42
Oct 30 04:50:16.322417 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:16.322388    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-57mpg"]
Oct 30 04:50:16.324963 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:50:16.324938    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0a25543c_936a_4521_b874_2ae3f9dcc61c.slice/crio-73b0d96609f504ac148e37c11d83ebe812cfda0d7d8045e4bdd3bf9598d2c9f5 WatchSource:0}: Error finding container 73b0d96609f504ac148e37c11d83ebe812cfda0d7d8045e4bdd3bf9598d2c9f5: Status 404 returned error can't find the container with id 73b0d96609f504ac148e37c11d83ebe812cfda0d7d8045e4bdd3bf9598d2c9f5
Oct 30 04:50:16.372708 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:16.372683    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-t2c9k"]
Oct 30 04:50:16.375545 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:50:16.375519    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a337a14_d243_45dc_8131_0d00b38564e8.slice/crio-e712089dc86acf26dd5b41c432096539afe47fa3c7714f3489dafe9a2bb4fa03 WatchSource:0}: Error finding container e712089dc86acf26dd5b41c432096539afe47fa3c7714f3489dafe9a2bb4fa03: Status 404 returned error can't find the container with id e712089dc86acf26dd5b41c432096539afe47fa3c7714f3489dafe9a2bb4fa03
Oct 30 04:50:17.206288 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:17.206247    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-57mpg" event={"ID":"0a25543c-936a-4521-b874-2ae3f9dcc61c","Type":"ContainerStarted","Data":"73b0d96609f504ac148e37c11d83ebe812cfda0d7d8045e4bdd3bf9598d2c9f5"}
Oct 30 04:50:17.208050 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:17.207990    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-4jgtx" event={"ID":"c712ba13-198b-48ba-9464-8b561f465646","Type":"ContainerStarted","Data":"e1eda833f5258a050f99ca9e9714747b882f2eb0918911ede0180da7facc0d42"}
Oct 30 04:50:17.210311 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:17.210281    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-t2c9k" event={"ID":"6a337a14-d243-45dc-8131-0d00b38564e8","Type":"ContainerStarted","Data":"e712089dc86acf26dd5b41c432096539afe47fa3c7714f3489dafe9a2bb4fa03"}
Oct 30 04:50:17.916571 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:17.916528    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:50:18.018572 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:18.018492    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:50:18.116535 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:18.116483    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:50:18.217241 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:18.217141    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:50:18.316912 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:18.316876    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:50:19.221432 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:19.221399    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-57mpg" event={"ID":"0a25543c-936a-4521-b874-2ae3f9dcc61c","Type":"ContainerStarted","Data":"91fa7371a9a1c85d4ff068605ac5613d4b298e8fe6ed02a0ec676ad8f5bd9637"}
Oct 30 04:50:19.221862 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:19.221503    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-57mpg"
Oct 30 04:50:19.223055 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:19.223033    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-4jgtx" event={"ID":"c712ba13-198b-48ba-9464-8b561f465646","Type":"ContainerStarted","Data":"372b37b16d8becfa3b607a656cce6197dbf6681bf65957df1f406abe11a99704"}
Oct 30 04:50:19.223177 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:19.223092    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-4jgtx"
Oct 30 04:50:19.224481 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:19.224462    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-t2c9k" event={"ID":"6a337a14-d243-45dc-8131-0d00b38564e8","Type":"ContainerStarted","Data":"82c6de3528e03dccd8e3fa63d435a1990973e14725e5abcf63729aac83c38d2d"}
Oct 30 04:50:19.224574 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:19.224500    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-t2c9k"
Oct 30 04:50:19.258150 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:19.258106    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-57mpg" podStartSLOduration=1.9057824349999999 podStartE2EDuration="4.258093964s" totalImagesPullingTime="2.352311529s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:50:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:50:19.257222605 +0000 UTC m=+348.481042523" watchObservedRunningTime="2026-10-30 04:50:19.258093964 +0000 UTC m=+348.481913861"
Oct 30 04:50:19.293576 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:19.293532    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-t2c9k" podStartSLOduration=2.391245645 podStartE2EDuration="4.293520307s" totalImagesPullingTime="1.902274662s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:50:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:50:19.292621961 +0000 UTC m=+348.516441861" watchObservedRunningTime="2026-10-30 04:50:19.293520307 +0000 UTC m=+348.517340208"
Oct 30 04:50:19.328660 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:19.328612    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-4jgtx" podStartSLOduration=1.902714059 podStartE2EDuration="4.328599644s" totalImagesPullingTime="2.425885585s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:50:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:50:19.326961555 +0000 UTC m=+348.550781467" watchObservedRunningTime="2026-10-30 04:50:19.328599644 +0000 UTC m=+348.552419544"
Oct 30 04:50:25.928446 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:25.928402    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:50:26.082668 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:26.082622    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:50:26.817208 ip-10-0-135-98 kubenswrapper[5898]: E1030 04:50:26.817169    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 04:50:30.233242 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:30.233216    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-t2c9k"
Oct 30 04:50:37.197974 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:37.197938    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:50:50.231660 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:50.231579    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-57mpg"
Oct 30 04:50:50.235189 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:50:50.235172    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-4jgtx"
Oct 30 04:50:56.816519 ip-10-0-135-98 kubenswrapper[5898]: E1030 04:50:56.816479    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 04:51:05.803048 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:05.803017    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:51:05.902711 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:05.902681    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:51:07.924056 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:07.924024    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vd4wn"]
Oct 30 04:51:07.927797 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:07.927765    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vd4wn"
Oct 30 04:51:07.942288 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:07.942133    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-5r6zn\""
Oct 30 04:51:07.953863 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:07.953835    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 04:51:07.954001 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:07.953872    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 04:51:07.955175 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:07.955156    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vd4wn"]
Oct 30 04:51:08.020945 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:08.020913    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am4wnl"]
Oct 30 04:51:08.024479 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:08.024460    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am4wnl"
Oct 30 04:51:08.029696 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:08.029676    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tnnm\" (UniqueName: \"kubernetes.io/projected/3ca1f3dc-1061-408b-bfc1-fcdef4dc7935-kube-api-access-9tnnm\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vd4wn\" (UID: \"3ca1f3dc-1061-408b-bfc1-fcdef4dc7935\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vd4wn"
Oct 30 04:51:08.029832 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:08.029706    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3ca1f3dc-1061-408b-bfc1-fcdef4dc7935-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vd4wn\" (UID: \"3ca1f3dc-1061-408b-bfc1-fcdef4dc7935\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vd4wn"
Oct 30 04:51:08.029832 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:08.029734    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3ca1f3dc-1061-408b-bfc1-fcdef4dc7935-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vd4wn\" (UID: \"3ca1f3dc-1061-408b-bfc1-fcdef4dc7935\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vd4wn"
Oct 30 04:51:08.042134 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:08.042110    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am4wnl"]
Oct 30 04:51:08.043806 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:08.043784    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-cn978\""
Oct 30 04:51:08.115222 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:08.115187    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nngl4"]
Oct 30 04:51:08.118948 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:08.118927    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nngl4"
Oct 30 04:51:08.130557 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:08.130535    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0b56bd2e-ac97-4ef8-a6a1-ce0e13f26a50-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am4wnl\" (UID: \"0b56bd2e-ac97-4ef8-a6a1-ce0e13f26a50\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am4wnl"
Oct 30 04:51:08.130670 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:08.130573    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3ca1f3dc-1061-408b-bfc1-fcdef4dc7935-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vd4wn\" (UID: \"3ca1f3dc-1061-408b-bfc1-fcdef4dc7935\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vd4wn"
Oct 30 04:51:08.130737 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:08.130720    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0b56bd2e-ac97-4ef8-a6a1-ce0e13f26a50-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am4wnl\" (UID: \"0b56bd2e-ac97-4ef8-a6a1-ce0e13f26a50\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am4wnl"
Oct 30 04:51:08.130840 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:08.130823    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tnnm\" (UniqueName: \"kubernetes.io/projected/3ca1f3dc-1061-408b-bfc1-fcdef4dc7935-kube-api-access-9tnnm\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vd4wn\" (UID: \"3ca1f3dc-1061-408b-bfc1-fcdef4dc7935\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vd4wn"
Oct 30 04:51:08.130919 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:08.130849    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3ca1f3dc-1061-408b-bfc1-fcdef4dc7935-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vd4wn\" (UID: \"3ca1f3dc-1061-408b-bfc1-fcdef4dc7935\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vd4wn"
Oct 30 04:51:08.130989 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:08.130920    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nblgg\" (UniqueName: \"kubernetes.io/projected/0b56bd2e-ac97-4ef8-a6a1-ce0e13f26a50-kube-api-access-nblgg\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am4wnl\" (UID: \"0b56bd2e-ac97-4ef8-a6a1-ce0e13f26a50\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am4wnl"
Oct 30 04:51:08.131040 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:08.130978    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3ca1f3dc-1061-408b-bfc1-fcdef4dc7935-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vd4wn\" (UID: \"3ca1f3dc-1061-408b-bfc1-fcdef4dc7935\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vd4wn"
Oct 30 04:51:08.131175 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:08.131159    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3ca1f3dc-1061-408b-bfc1-fcdef4dc7935-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vd4wn\" (UID: \"3ca1f3dc-1061-408b-bfc1-fcdef4dc7935\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vd4wn"
Oct 30 04:51:08.133831 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:08.133810    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nngl4"]
Oct 30 04:51:08.143942 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:08.143922    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-d4mqp\""
Oct 30 04:51:08.149570 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:08.149550    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tnnm\" (UniqueName: \"kubernetes.io/projected/3ca1f3dc-1061-408b-bfc1-fcdef4dc7935-kube-api-access-9tnnm\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vd4wn\" (UID: \"3ca1f3dc-1061-408b-bfc1-fcdef4dc7935\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vd4wn"
Oct 30 04:51:08.229245 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:08.229175    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c26ht2"]
Oct 30 04:51:08.231327 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:08.231301    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/009597ee-ffd4-4595-9345-817bfda51ac0-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nngl4\" (UID: \"009597ee-ffd4-4595-9345-817bfda51ac0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nngl4"
Oct 30 04:51:08.231459 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:08.231349    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/009597ee-ffd4-4595-9345-817bfda51ac0-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nngl4\" (UID: \"009597ee-ffd4-4595-9345-817bfda51ac0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nngl4"
Oct 30 04:51:08.231459 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:08.231424    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0b56bd2e-ac97-4ef8-a6a1-ce0e13f26a50-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am4wnl\" (UID: \"0b56bd2e-ac97-4ef8-a6a1-ce0e13f26a50\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am4wnl"
Oct 30 04:51:08.231564 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:08.231492    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q6tg\" (UniqueName: \"kubernetes.io/projected/009597ee-ffd4-4595-9345-817bfda51ac0-kube-api-access-6q6tg\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nngl4\" (UID: \"009597ee-ffd4-4595-9345-817bfda51ac0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nngl4"
Oct 30 04:51:08.231608 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:08.231595    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nblgg\" (UniqueName: \"kubernetes.io/projected/0b56bd2e-ac97-4ef8-a6a1-ce0e13f26a50-kube-api-access-nblgg\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am4wnl\" (UID: \"0b56bd2e-ac97-4ef8-a6a1-ce0e13f26a50\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am4wnl"
Oct 30 04:51:08.231643 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:08.231617    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0b56bd2e-ac97-4ef8-a6a1-ce0e13f26a50-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am4wnl\" (UID: \"0b56bd2e-ac97-4ef8-a6a1-ce0e13f26a50\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am4wnl"
Oct 30 04:51:08.231727 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:08.231711    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0b56bd2e-ac97-4ef8-a6a1-ce0e13f26a50-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am4wnl\" (UID: \"0b56bd2e-ac97-4ef8-a6a1-ce0e13f26a50\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am4wnl"
Oct 30 04:51:08.231863 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:08.231847    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0b56bd2e-ac97-4ef8-a6a1-ce0e13f26a50-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am4wnl\" (UID: \"0b56bd2e-ac97-4ef8-a6a1-ce0e13f26a50\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am4wnl"
Oct 30 04:51:08.232722 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:08.232709    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c26ht2"
Oct 30 04:51:08.243217 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:08.243197    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vd4wn"
Oct 30 04:51:08.249498 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:08.249475    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-hlvcp\""
Oct 30 04:51:08.251657 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:08.251631    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c26ht2"]
Oct 30 04:51:08.253624 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:08.253601    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nblgg\" (UniqueName: \"kubernetes.io/projected/0b56bd2e-ac97-4ef8-a6a1-ce0e13f26a50-kube-api-access-nblgg\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am4wnl\" (UID: \"0b56bd2e-ac97-4ef8-a6a1-ce0e13f26a50\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am4wnl"
Oct 30 04:51:08.332207 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:08.332177    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/009597ee-ffd4-4595-9345-817bfda51ac0-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nngl4\" (UID: \"009597ee-ffd4-4595-9345-817bfda51ac0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nngl4"
Oct 30 04:51:08.332376 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:08.332224    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb427\" (UniqueName: \"kubernetes.io/projected/4b29d472-3538-447e-a4f5-62002932665f-kube-api-access-rb427\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c26ht2\" (UID: \"4b29d472-3538-447e-a4f5-62002932665f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c26ht2"
Oct 30 04:51:08.332376 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:08.332245    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/009597ee-ffd4-4595-9345-817bfda51ac0-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nngl4\" (UID: \"009597ee-ffd4-4595-9345-817bfda51ac0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nngl4"
Oct 30 04:51:08.332376 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:08.332349    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4b29d472-3538-447e-a4f5-62002932665f-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c26ht2\" (UID: \"4b29d472-3538-447e-a4f5-62002932665f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c26ht2"
Oct 30 04:51:08.332539 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:08.332439    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q6tg\" (UniqueName: \"kubernetes.io/projected/009597ee-ffd4-4595-9345-817bfda51ac0-kube-api-access-6q6tg\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nngl4\" (UID: \"009597ee-ffd4-4595-9345-817bfda51ac0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nngl4"
Oct 30 04:51:08.332539 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:08.332508    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4b29d472-3538-447e-a4f5-62002932665f-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c26ht2\" (UID: \"4b29d472-3538-447e-a4f5-62002932665f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c26ht2"
Oct 30 04:51:08.332729 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:08.332701    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/009597ee-ffd4-4595-9345-817bfda51ac0-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nngl4\" (UID: \"009597ee-ffd4-4595-9345-817bfda51ac0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nngl4"
Oct 30 04:51:08.332729 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:08.332719    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/009597ee-ffd4-4595-9345-817bfda51ac0-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nngl4\" (UID: \"009597ee-ffd4-4595-9345-817bfda51ac0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nngl4"
Oct 30 04:51:08.340119 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:08.340102    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am4wnl"
Oct 30 04:51:08.351348 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:08.351326    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q6tg\" (UniqueName: \"kubernetes.io/projected/009597ee-ffd4-4595-9345-817bfda51ac0-kube-api-access-6q6tg\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nngl4\" (UID: \"009597ee-ffd4-4595-9345-817bfda51ac0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nngl4"
Oct 30 04:51:08.433128 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:08.433101    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4b29d472-3538-447e-a4f5-62002932665f-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c26ht2\" (UID: \"4b29d472-3538-447e-a4f5-62002932665f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c26ht2"
Oct 30 04:51:08.433244 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:08.433147    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb427\" (UniqueName: \"kubernetes.io/projected/4b29d472-3538-447e-a4f5-62002932665f-kube-api-access-rb427\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c26ht2\" (UID: \"4b29d472-3538-447e-a4f5-62002932665f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c26ht2"
Oct 30 04:51:08.433244 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:08.433195    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4b29d472-3538-447e-a4f5-62002932665f-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c26ht2\" (UID: \"4b29d472-3538-447e-a4f5-62002932665f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c26ht2"
Oct 30 04:51:08.433547 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:08.433527    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4b29d472-3538-447e-a4f5-62002932665f-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c26ht2\" (UID: \"4b29d472-3538-447e-a4f5-62002932665f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c26ht2"
Oct 30 04:51:08.433587 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:08.433566    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4b29d472-3538-447e-a4f5-62002932665f-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c26ht2\" (UID: \"4b29d472-3538-447e-a4f5-62002932665f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c26ht2"
Oct 30 04:51:08.435013 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:08.434994    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nngl4"
Oct 30 04:51:08.451959 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:08.451933    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb427\" (UniqueName: \"kubernetes.io/projected/4b29d472-3538-447e-a4f5-62002932665f-kube-api-access-rb427\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c26ht2\" (UID: \"4b29d472-3538-447e-a4f5-62002932665f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c26ht2"
Oct 30 04:51:08.535067 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:08.535030    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vd4wn"]
Oct 30 04:51:08.539818 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:51:08.539789    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ca1f3dc_1061_408b_bfc1_fcdef4dc7935.slice/crio-6b04d7b3ede6ba1f3f8366f63db32fd0cd69bafd369bb88d539cd7e8e5af5e53 WatchSource:0}: Error finding container 6b04d7b3ede6ba1f3f8366f63db32fd0cd69bafd369bb88d539cd7e8e5af5e53: Status 404 returned error can't find the container with id 6b04d7b3ede6ba1f3f8366f63db32fd0cd69bafd369bb88d539cd7e8e5af5e53
Oct 30 04:51:08.549288 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:08.549264    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c26ht2"
Oct 30 04:51:08.633646 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:08.633565    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am4wnl"]
Oct 30 04:51:08.636500 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:51:08.636471    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b56bd2e_ac97_4ef8_a6a1_ce0e13f26a50.slice/crio-9129a7ea68921e533de94a10da9c2687b0502807773cc105b4184cbb282ef2c5 WatchSource:0}: Error finding container 9129a7ea68921e533de94a10da9c2687b0502807773cc105b4184cbb282ef2c5: Status 404 returned error can't find the container with id 9129a7ea68921e533de94a10da9c2687b0502807773cc105b4184cbb282ef2c5
Oct 30 04:51:08.726655 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:08.726628    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nngl4"]
Oct 30 04:51:08.729860 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:51:08.729838    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod009597ee_ffd4_4595_9345_817bfda51ac0.slice/crio-2e23ccf349a1706f26c15d29eef6515fd4d52395f17bfa1c69147798959cd88d WatchSource:0}: Error finding container 2e23ccf349a1706f26c15d29eef6515fd4d52395f17bfa1c69147798959cd88d: Status 404 returned error can't find the container with id 2e23ccf349a1706f26c15d29eef6515fd4d52395f17bfa1c69147798959cd88d
Oct 30 04:51:08.844119 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:08.844091    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c26ht2"]
Oct 30 04:51:08.888614 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:51:08.888577    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b29d472_3538_447e_a4f5_62002932665f.slice/crio-f71bc134d64b6db412c7b4db0429ececd9bfb96070617f700db29b8e311c1305 WatchSource:0}: Error finding container f71bc134d64b6db412c7b4db0429ececd9bfb96070617f700db29b8e311c1305: Status 404 returned error can't find the container with id f71bc134d64b6db412c7b4db0429ececd9bfb96070617f700db29b8e311c1305
Oct 30 04:51:09.438041 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:09.438008    5898 generic.go:356] "Generic (PLEG): container finished" podID="0b56bd2e-ac97-4ef8-a6a1-ce0e13f26a50" containerID="9f988fbc02037667e2e6c837b235a8ac9c2218b98b91352cd23e1e824b085f23" exitCode=0
Oct 30 04:51:09.438474 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:09.438088    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am4wnl" event={"ID":"0b56bd2e-ac97-4ef8-a6a1-ce0e13f26a50","Type":"ContainerDied","Data":"9f988fbc02037667e2e6c837b235a8ac9c2218b98b91352cd23e1e824b085f23"}
Oct 30 04:51:09.438474 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:09.438131    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am4wnl" event={"ID":"0b56bd2e-ac97-4ef8-a6a1-ce0e13f26a50","Type":"ContainerStarted","Data":"9129a7ea68921e533de94a10da9c2687b0502807773cc105b4184cbb282ef2c5"}
Oct 30 04:51:09.439688 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:09.439658    5898 generic.go:356] "Generic (PLEG): container finished" podID="4b29d472-3538-447e-a4f5-62002932665f" containerID="0edc3ef38f57ae4559d536f678451f98414c70a9e7384b113e27d6072b868be0" exitCode=0
Oct 30 04:51:09.440010 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:09.439742    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c26ht2" event={"ID":"4b29d472-3538-447e-a4f5-62002932665f","Type":"ContainerDied","Data":"0edc3ef38f57ae4559d536f678451f98414c70a9e7384b113e27d6072b868be0"}
Oct 30 04:51:09.440010 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:09.439793    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c26ht2" event={"ID":"4b29d472-3538-447e-a4f5-62002932665f","Type":"ContainerStarted","Data":"f71bc134d64b6db412c7b4db0429ececd9bfb96070617f700db29b8e311c1305"}
Oct 30 04:51:09.443666 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:09.443644    5898 generic.go:356] "Generic (PLEG): container finished" podID="3ca1f3dc-1061-408b-bfc1-fcdef4dc7935" containerID="d51b564fa16293770bdfc615157129b278ffaffb00a918f83fa12cd3dcff4119" exitCode=0
Oct 30 04:51:09.443814 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:09.443758    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vd4wn" event={"ID":"3ca1f3dc-1061-408b-bfc1-fcdef4dc7935","Type":"ContainerDied","Data":"d51b564fa16293770bdfc615157129b278ffaffb00a918f83fa12cd3dcff4119"}
Oct 30 04:51:09.443814 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:09.443807    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vd4wn" event={"ID":"3ca1f3dc-1061-408b-bfc1-fcdef4dc7935","Type":"ContainerStarted","Data":"6b04d7b3ede6ba1f3f8366f63db32fd0cd69bafd369bb88d539cd7e8e5af5e53"}
Oct 30 04:51:09.445841 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:09.445822    5898 generic.go:356] "Generic (PLEG): container finished" podID="009597ee-ffd4-4595-9345-817bfda51ac0" containerID="b0492a4de7cadac7839ca4e2aa3f46d8b17c0ecc77ffbe6af7c3109e22e3b451" exitCode=0
Oct 30 04:51:09.445961 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:09.445848    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nngl4" event={"ID":"009597ee-ffd4-4595-9345-817bfda51ac0","Type":"ContainerDied","Data":"b0492a4de7cadac7839ca4e2aa3f46d8b17c0ecc77ffbe6af7c3109e22e3b451"}
Oct 30 04:51:09.445961 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:09.445879    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nngl4" event={"ID":"009597ee-ffd4-4595-9345-817bfda51ac0","Type":"ContainerStarted","Data":"2e23ccf349a1706f26c15d29eef6515fd4d52395f17bfa1c69147798959cd88d"}
Oct 30 04:51:10.452040 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:10.451983    5898 generic.go:356] "Generic (PLEG): container finished" podID="0b56bd2e-ac97-4ef8-a6a1-ce0e13f26a50" containerID="e00d861329361218671a8074331bd452fbf7c3a8921cec29039f8be8d70484c4" exitCode=0
Oct 30 04:51:10.452382 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:10.452062    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am4wnl" event={"ID":"0b56bd2e-ac97-4ef8-a6a1-ce0e13f26a50","Type":"ContainerDied","Data":"e00d861329361218671a8074331bd452fbf7c3a8921cec29039f8be8d70484c4"}
Oct 30 04:51:10.453998 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:10.453978    5898 generic.go:356] "Generic (PLEG): container finished" podID="4b29d472-3538-447e-a4f5-62002932665f" containerID="2ac19eb9063c321d8492f11fe8fcee87caf721f17c327ce01c602043e26483f9" exitCode=0
Oct 30 04:51:10.454100 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:10.454038    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c26ht2" event={"ID":"4b29d472-3538-447e-a4f5-62002932665f","Type":"ContainerDied","Data":"2ac19eb9063c321d8492f11fe8fcee87caf721f17c327ce01c602043e26483f9"}
Oct 30 04:51:10.456085 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:10.456063    5898 generic.go:356] "Generic (PLEG): container finished" podID="3ca1f3dc-1061-408b-bfc1-fcdef4dc7935" containerID="56e1f503719bbd8659d73aa46d345477e98aa616d4d9706b45dbdaa6ec533fa3" exitCode=0
Oct 30 04:51:10.456173 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:10.456106    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vd4wn" event={"ID":"3ca1f3dc-1061-408b-bfc1-fcdef4dc7935","Type":"ContainerDied","Data":"56e1f503719bbd8659d73aa46d345477e98aa616d4d9706b45dbdaa6ec533fa3"}
Oct 30 04:51:11.462177 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:11.462146    5898 generic.go:356] "Generic (PLEG): container finished" podID="009597ee-ffd4-4595-9345-817bfda51ac0" containerID="020aeca0786d5f789a54794cd40c272aa8dfaca94211c10783d49b2db0511317" exitCode=0
Oct 30 04:51:11.462607 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:11.462230    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nngl4" event={"ID":"009597ee-ffd4-4595-9345-817bfda51ac0","Type":"ContainerDied","Data":"020aeca0786d5f789a54794cd40c272aa8dfaca94211c10783d49b2db0511317"}
Oct 30 04:51:11.464775 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:11.464746    5898 generic.go:356] "Generic (PLEG): container finished" podID="0b56bd2e-ac97-4ef8-a6a1-ce0e13f26a50" containerID="d640c48a453474fc094a11e43aec37708c4afa7e52eff022686fc274c4ee82c9" exitCode=0
Oct 30 04:51:11.464925 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:11.464778    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am4wnl" event={"ID":"0b56bd2e-ac97-4ef8-a6a1-ce0e13f26a50","Type":"ContainerDied","Data":"d640c48a453474fc094a11e43aec37708c4afa7e52eff022686fc274c4ee82c9"}
Oct 30 04:51:11.466808 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:11.466790    5898 generic.go:356] "Generic (PLEG): container finished" podID="4b29d472-3538-447e-a4f5-62002932665f" containerID="77983f6750b19fb10b4b6399315cc63f2396a1f76261087cf49cd586bbb3fcdd" exitCode=0
Oct 30 04:51:11.466923 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:11.466856    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c26ht2" event={"ID":"4b29d472-3538-447e-a4f5-62002932665f","Type":"ContainerDied","Data":"77983f6750b19fb10b4b6399315cc63f2396a1f76261087cf49cd586bbb3fcdd"}
Oct 30 04:51:11.468832 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:11.468813    5898 generic.go:356] "Generic (PLEG): container finished" podID="3ca1f3dc-1061-408b-bfc1-fcdef4dc7935" containerID="63a305c4009c125c412c4ab461356c1f21b9e5abdbc188a93bed91788d84f745" exitCode=0
Oct 30 04:51:11.468940 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:11.468838    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vd4wn" event={"ID":"3ca1f3dc-1061-408b-bfc1-fcdef4dc7935","Type":"ContainerDied","Data":"63a305c4009c125c412c4ab461356c1f21b9e5abdbc188a93bed91788d84f745"}
Oct 30 04:51:12.475755 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:12.475715    5898 generic.go:356] "Generic (PLEG): container finished" podID="009597ee-ffd4-4595-9345-817bfda51ac0" containerID="60925288ba956249ea9fc4a9d73f9155e69f5d77dd63d9af5c6986e0b87f2389" exitCode=0
Oct 30 04:51:12.476145 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:12.475801    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nngl4" event={"ID":"009597ee-ffd4-4595-9345-817bfda51ac0","Type":"ContainerDied","Data":"60925288ba956249ea9fc4a9d73f9155e69f5d77dd63d9af5c6986e0b87f2389"}
Oct 30 04:51:12.625949 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:12.625896    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am4wnl"
Oct 30 04:51:12.646477 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:12.646456    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c26ht2"
Oct 30 04:51:12.666466 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:12.666437    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b56bd2e-ac97-4ef8-a6a1-ce0e13f26a50-util\" (UniqueName: \"kubernetes.io/empty-dir/0b56bd2e-ac97-4ef8-a6a1-ce0e13f26a50-util\") pod \"0b56bd2e-ac97-4ef8-a6a1-ce0e13f26a50\" (UID: \"0b56bd2e-ac97-4ef8-a6a1-ce0e13f26a50\") "
Oct 30 04:51:12.666565 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:12.666524    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b56bd2e-ac97-4ef8-a6a1-ce0e13f26a50-kube-api-access-nblgg\" (UniqueName: \"kubernetes.io/projected/0b56bd2e-ac97-4ef8-a6a1-ce0e13f26a50-kube-api-access-nblgg\") pod \"0b56bd2e-ac97-4ef8-a6a1-ce0e13f26a50\" (UID: \"0b56bd2e-ac97-4ef8-a6a1-ce0e13f26a50\") "
Oct 30 04:51:12.666609 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:12.666581    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b56bd2e-ac97-4ef8-a6a1-ce0e13f26a50-bundle\" (UniqueName: \"kubernetes.io/empty-dir/0b56bd2e-ac97-4ef8-a6a1-ce0e13f26a50-bundle\") pod \"0b56bd2e-ac97-4ef8-a6a1-ce0e13f26a50\" (UID: \"0b56bd2e-ac97-4ef8-a6a1-ce0e13f26a50\") "
Oct 30 04:51:12.667408 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:12.667385    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vd4wn"
Oct 30 04:51:12.667611 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:12.667583    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b56bd2e-ac97-4ef8-a6a1-ce0e13f26a50-bundle" pod "0b56bd2e-ac97-4ef8-a6a1-ce0e13f26a50" (UID: "0b56bd2e-ac97-4ef8-a6a1-ce0e13f26a50"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:51:12.668696 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:12.668669    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b56bd2e-ac97-4ef8-a6a1-ce0e13f26a50-kube-api-access-nblgg" pod "0b56bd2e-ac97-4ef8-a6a1-ce0e13f26a50" (UID: "0b56bd2e-ac97-4ef8-a6a1-ce0e13f26a50"). InnerVolumeSpecName "kube-api-access-nblgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:51:12.671859 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:12.671836    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b56bd2e-ac97-4ef8-a6a1-ce0e13f26a50-util" pod "0b56bd2e-ac97-4ef8-a6a1-ce0e13f26a50" (UID: "0b56bd2e-ac97-4ef8-a6a1-ce0e13f26a50"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:51:12.767454 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:12.767390    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b29d472-3538-447e-a4f5-62002932665f-kube-api-access-rb427\" (UniqueName: \"kubernetes.io/projected/4b29d472-3538-447e-a4f5-62002932665f-kube-api-access-rb427\") pod \"4b29d472-3538-447e-a4f5-62002932665f\" (UID: \"4b29d472-3538-447e-a4f5-62002932665f\") "
Oct 30 04:51:12.767454 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:12.767448    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b29d472-3538-447e-a4f5-62002932665f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/4b29d472-3538-447e-a4f5-62002932665f-bundle\") pod \"4b29d472-3538-447e-a4f5-62002932665f\" (UID: \"4b29d472-3538-447e-a4f5-62002932665f\") "
Oct 30 04:51:12.767624 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:12.767468    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ca1f3dc-1061-408b-bfc1-fcdef4dc7935-kube-api-access-9tnnm\" (UniqueName: \"kubernetes.io/projected/3ca1f3dc-1061-408b-bfc1-fcdef4dc7935-kube-api-access-9tnnm\") pod \"3ca1f3dc-1061-408b-bfc1-fcdef4dc7935\" (UID: \"3ca1f3dc-1061-408b-bfc1-fcdef4dc7935\") "
Oct 30 04:51:12.767624 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:12.767520    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ca1f3dc-1061-408b-bfc1-fcdef4dc7935-bundle\" (UniqueName: \"kubernetes.io/empty-dir/3ca1f3dc-1061-408b-bfc1-fcdef4dc7935-bundle\") pod \"3ca1f3dc-1061-408b-bfc1-fcdef4dc7935\" (UID: \"3ca1f3dc-1061-408b-bfc1-fcdef4dc7935\") "
Oct 30 04:51:12.767624 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:12.767561    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ca1f3dc-1061-408b-bfc1-fcdef4dc7935-util\" (UniqueName: \"kubernetes.io/empty-dir/3ca1f3dc-1061-408b-bfc1-fcdef4dc7935-util\") pod \"3ca1f3dc-1061-408b-bfc1-fcdef4dc7935\" (UID: \"3ca1f3dc-1061-408b-bfc1-fcdef4dc7935\") "
Oct 30 04:51:12.767624 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:12.767580    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b29d472-3538-447e-a4f5-62002932665f-util\" (UniqueName: \"kubernetes.io/empty-dir/4b29d472-3538-447e-a4f5-62002932665f-util\") pod \"4b29d472-3538-447e-a4f5-62002932665f\" (UID: \"4b29d472-3538-447e-a4f5-62002932665f\") "
Oct 30 04:51:12.767818 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:12.767790    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nblgg\" (UniqueName: \"kubernetes.io/projected/0b56bd2e-ac97-4ef8-a6a1-ce0e13f26a50-kube-api-access-nblgg\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 04:51:12.767933 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:12.767828    5898 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0b56bd2e-ac97-4ef8-a6a1-ce0e13f26a50-bundle\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 04:51:12.767933 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:12.767843    5898 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0b56bd2e-ac97-4ef8-a6a1-ce0e13f26a50-util\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 04:51:12.768262 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:12.768236    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ca1f3dc-1061-408b-bfc1-fcdef4dc7935-bundle" pod "3ca1f3dc-1061-408b-bfc1-fcdef4dc7935" (UID: "3ca1f3dc-1061-408b-bfc1-fcdef4dc7935"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:51:12.768262 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:12.768243    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b29d472-3538-447e-a4f5-62002932665f-bundle" pod "4b29d472-3538-447e-a4f5-62002932665f" (UID: "4b29d472-3538-447e-a4f5-62002932665f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:51:12.769624 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:12.769592    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b29d472-3538-447e-a4f5-62002932665f-kube-api-access-rb427" pod "4b29d472-3538-447e-a4f5-62002932665f" (UID: "4b29d472-3538-447e-a4f5-62002932665f"). InnerVolumeSpecName "kube-api-access-rb427". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:51:12.769867 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:12.769850    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ca1f3dc-1061-408b-bfc1-fcdef4dc7935-kube-api-access-9tnnm" pod "3ca1f3dc-1061-408b-bfc1-fcdef4dc7935" (UID: "3ca1f3dc-1061-408b-bfc1-fcdef4dc7935"). InnerVolumeSpecName "kube-api-access-9tnnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:51:12.773710 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:12.773685    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b29d472-3538-447e-a4f5-62002932665f-util" pod "4b29d472-3538-447e-a4f5-62002932665f" (UID: "4b29d472-3538-447e-a4f5-62002932665f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:51:12.774169 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:12.774146    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ca1f3dc-1061-408b-bfc1-fcdef4dc7935-util" pod "3ca1f3dc-1061-408b-bfc1-fcdef4dc7935" (UID: "3ca1f3dc-1061-408b-bfc1-fcdef4dc7935"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:51:12.868487 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:12.868448    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb427\" (UniqueName: \"kubernetes.io/projected/4b29d472-3538-447e-a4f5-62002932665f-kube-api-access-rb427\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 04:51:12.868487 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:12.868486    5898 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4b29d472-3538-447e-a4f5-62002932665f-bundle\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 04:51:12.868487 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:12.868497    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tnnm\" (UniqueName: \"kubernetes.io/projected/3ca1f3dc-1061-408b-bfc1-fcdef4dc7935-kube-api-access-9tnnm\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 04:51:12.868687 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:12.868507    5898 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3ca1f3dc-1061-408b-bfc1-fcdef4dc7935-bundle\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 04:51:12.868687 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:12.868516    5898 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3ca1f3dc-1061-408b-bfc1-fcdef4dc7935-util\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 04:51:12.868687 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:12.868523    5898 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4b29d472-3538-447e-a4f5-62002932665f-util\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 04:51:13.483474 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:13.483447    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am4wnl"
Oct 30 04:51:13.483934 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:13.483439    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am4wnl" event={"ID":"0b56bd2e-ac97-4ef8-a6a1-ce0e13f26a50","Type":"ContainerDied","Data":"9129a7ea68921e533de94a10da9c2687b0502807773cc105b4184cbb282ef2c5"}
Oct 30 04:51:13.483934 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:13.483580    5898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9129a7ea68921e533de94a10da9c2687b0502807773cc105b4184cbb282ef2c5"
Oct 30 04:51:13.485569 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:13.485542    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c26ht2" event={"ID":"4b29d472-3538-447e-a4f5-62002932665f","Type":"ContainerDied","Data":"f71bc134d64b6db412c7b4db0429ececd9bfb96070617f700db29b8e311c1305"}
Oct 30 04:51:13.485569 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:13.485562    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c26ht2"
Oct 30 04:51:13.485771 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:13.485583    5898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f71bc134d64b6db412c7b4db0429ececd9bfb96070617f700db29b8e311c1305"
Oct 30 04:51:13.487615 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:13.487594    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vd4wn"
Oct 30 04:51:13.487738 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:13.487620    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vd4wn" event={"ID":"3ca1f3dc-1061-408b-bfc1-fcdef4dc7935","Type":"ContainerDied","Data":"6b04d7b3ede6ba1f3f8366f63db32fd0cd69bafd369bb88d539cd7e8e5af5e53"}
Oct 30 04:51:13.487738 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:13.487652    5898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b04d7b3ede6ba1f3f8366f63db32fd0cd69bafd369bb88d539cd7e8e5af5e53"
Oct 30 04:51:13.597310 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:13.597288    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nngl4"
Oct 30 04:51:13.673839 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:13.673810    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/009597ee-ffd4-4595-9345-817bfda51ac0-util\" (UniqueName: \"kubernetes.io/empty-dir/009597ee-ffd4-4595-9345-817bfda51ac0-util\") pod \"009597ee-ffd4-4595-9345-817bfda51ac0\" (UID: \"009597ee-ffd4-4595-9345-817bfda51ac0\") "
Oct 30 04:51:13.673982 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:13.673845    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/009597ee-ffd4-4595-9345-817bfda51ac0-bundle\" (UniqueName: \"kubernetes.io/empty-dir/009597ee-ffd4-4595-9345-817bfda51ac0-bundle\") pod \"009597ee-ffd4-4595-9345-817bfda51ac0\" (UID: \"009597ee-ffd4-4595-9345-817bfda51ac0\") "
Oct 30 04:51:13.673982 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:13.673937    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/009597ee-ffd4-4595-9345-817bfda51ac0-kube-api-access-6q6tg\" (UniqueName: \"kubernetes.io/projected/009597ee-ffd4-4595-9345-817bfda51ac0-kube-api-access-6q6tg\") pod \"009597ee-ffd4-4595-9345-817bfda51ac0\" (UID: \"009597ee-ffd4-4595-9345-817bfda51ac0\") "
Oct 30 04:51:13.674359 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:13.674326    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/009597ee-ffd4-4595-9345-817bfda51ac0-bundle" pod "009597ee-ffd4-4595-9345-817bfda51ac0" (UID: "009597ee-ffd4-4595-9345-817bfda51ac0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:51:13.675941 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:13.675915    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/009597ee-ffd4-4595-9345-817bfda51ac0-kube-api-access-6q6tg" pod "009597ee-ffd4-4595-9345-817bfda51ac0" (UID: "009597ee-ffd4-4595-9345-817bfda51ac0"). InnerVolumeSpecName "kube-api-access-6q6tg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:51:13.679393 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:13.679374    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/009597ee-ffd4-4595-9345-817bfda51ac0-util" pod "009597ee-ffd4-4595-9345-817bfda51ac0" (UID: "009597ee-ffd4-4595-9345-817bfda51ac0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:51:13.774787 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:13.774723    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6q6tg\" (UniqueName: \"kubernetes.io/projected/009597ee-ffd4-4595-9345-817bfda51ac0-kube-api-access-6q6tg\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 04:51:13.774787 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:13.774747    5898 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/009597ee-ffd4-4595-9345-817bfda51ac0-util\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 04:51:13.774787 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:13.774757    5898 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/009597ee-ffd4-4595-9345-817bfda51ac0-bundle\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 04:51:14.501523 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:14.501489    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nngl4" event={"ID":"009597ee-ffd4-4595-9345-817bfda51ac0","Type":"ContainerDied","Data":"2e23ccf349a1706f26c15d29eef6515fd4d52395f17bfa1c69147798959cd88d"}
Oct 30 04:51:14.501523 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:14.501511    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nngl4"
Oct 30 04:51:14.502009 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:14.501558    5898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e23ccf349a1706f26c15d29eef6515fd4d52395f17bfa1c69147798959cd88d"
Oct 30 04:51:21.224916 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:21.224859    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:51:21.948548 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:21.948517    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:51:24.999411 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:24.999379    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:51:25.184280 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:25.184244    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:51:26.387054 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:26.387024    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:51:26.816212 ip-10-0-135-98 kubenswrapper[5898]: E1030 04:51:26.816129    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 04:51:27.577869 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:27.577841    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:51:27.813752 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:27.813717    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:51:29.594408 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:29.594374    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:51:29.896676 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:29.896589    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:51:29.996961 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:29.996928    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:51:30.358084 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:30.358049    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:51:30.641600 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:30.641518    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:51:34.782118 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:34.782083    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:51:34.879722 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:34.879685    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:51:35.433212 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:35.433177    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:51:35.512368 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:35.512338    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:51:37.313070 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:37.313034    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:51:37.770968 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:37.770934    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:51:37.872656 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:37.872624    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:51:41.447733 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:41.447692    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-fgtkv"]
Oct 30 04:51:41.448200 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:41.448084    5898 cpu_manager.go:414] "containerMap: removing container" podUID="009597ee-ffd4-4595-9345-817bfda51ac0" containerName="util"
Oct 30 04:51:41.448200 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:41.448103    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="009597ee-ffd4-4595-9345-817bfda51ac0" containerName="util"
Oct 30 04:51:41.448200 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:41.448119    5898 cpu_manager.go:414] "containerMap: removing container" podUID="3ca1f3dc-1061-408b-bfc1-fcdef4dc7935" containerName="extract"
Oct 30 04:51:41.448200 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:41.448126    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ca1f3dc-1061-408b-bfc1-fcdef4dc7935" containerName="extract"
Oct 30 04:51:41.448200 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:41.448136    5898 cpu_manager.go:414] "containerMap: removing container" podUID="009597ee-ffd4-4595-9345-817bfda51ac0" containerName="extract"
Oct 30 04:51:41.448200 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:41.448146    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="009597ee-ffd4-4595-9345-817bfda51ac0" containerName="extract"
Oct 30 04:51:41.448200 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:41.448175    5898 cpu_manager.go:414] "containerMap: removing container" podUID="4b29d472-3538-447e-a4f5-62002932665f" containerName="pull"
Oct 30 04:51:41.448200 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:41.448182    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b29d472-3538-447e-a4f5-62002932665f" containerName="pull"
Oct 30 04:51:41.448200 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:41.448195    5898 cpu_manager.go:414] "containerMap: removing container" podUID="0b56bd2e-ac97-4ef8-a6a1-ce0e13f26a50" containerName="util"
Oct 30 04:51:41.448200 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:41.448202    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b56bd2e-ac97-4ef8-a6a1-ce0e13f26a50" containerName="util"
Oct 30 04:51:41.448609 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:41.448216    5898 cpu_manager.go:414] "containerMap: removing container" podUID="0b56bd2e-ac97-4ef8-a6a1-ce0e13f26a50" containerName="extract"
Oct 30 04:51:41.448609 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:41.448223    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b56bd2e-ac97-4ef8-a6a1-ce0e13f26a50" containerName="extract"
Oct 30 04:51:41.448609 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:41.448233    5898 cpu_manager.go:414] "containerMap: removing container" podUID="4b29d472-3538-447e-a4f5-62002932665f" containerName="util"
Oct 30 04:51:41.448609 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:41.448239    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b29d472-3538-447e-a4f5-62002932665f" containerName="util"
Oct 30 04:51:41.448609 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:41.448251    5898 cpu_manager.go:414] "containerMap: removing container" podUID="009597ee-ffd4-4595-9345-817bfda51ac0" containerName="pull"
Oct 30 04:51:41.448609 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:41.448259    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="009597ee-ffd4-4595-9345-817bfda51ac0" containerName="pull"
Oct 30 04:51:41.448609 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:41.448280    5898 cpu_manager.go:414] "containerMap: removing container" podUID="3ca1f3dc-1061-408b-bfc1-fcdef4dc7935" containerName="util"
Oct 30 04:51:41.448609 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:41.448287    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ca1f3dc-1061-408b-bfc1-fcdef4dc7935" containerName="util"
Oct 30 04:51:41.448609 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:41.448296    5898 cpu_manager.go:414] "containerMap: removing container" podUID="0b56bd2e-ac97-4ef8-a6a1-ce0e13f26a50" containerName="pull"
Oct 30 04:51:41.448609 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:41.448303    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b56bd2e-ac97-4ef8-a6a1-ce0e13f26a50" containerName="pull"
Oct 30 04:51:41.448609 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:41.448309    5898 cpu_manager.go:414] "containerMap: removing container" podUID="4b29d472-3538-447e-a4f5-62002932665f" containerName="extract"
Oct 30 04:51:41.448609 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:41.448313    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b29d472-3538-447e-a4f5-62002932665f" containerName="extract"
Oct 30 04:51:41.448609 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:41.448320    5898 cpu_manager.go:414] "containerMap: removing container" podUID="3ca1f3dc-1061-408b-bfc1-fcdef4dc7935" containerName="pull"
Oct 30 04:51:41.448609 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:41.448327    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ca1f3dc-1061-408b-bfc1-fcdef4dc7935" containerName="pull"
Oct 30 04:51:41.448609 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:41.448414    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b56bd2e-ac97-4ef8-a6a1-ce0e13f26a50" containerName="extract"
Oct 30 04:51:41.448609 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:41.448423    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b29d472-3538-447e-a4f5-62002932665f" containerName="extract"
Oct 30 04:51:41.448609 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:41.448431    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="009597ee-ffd4-4595-9345-817bfda51ac0" containerName="extract"
Oct 30 04:51:41.448609 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:41.448437    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ca1f3dc-1061-408b-bfc1-fcdef4dc7935" containerName="extract"
Oct 30 04:51:41.452775 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:41.452750    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:51:41.453248 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:41.453233    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fgtkv"
Oct 30 04:51:41.467808 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:41.467782    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 04:51:41.469819 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:41.469797    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-fgtkv"]
Oct 30 04:51:41.479025 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:41.479005    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Oct 30 04:51:41.479121 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:41.479024    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Oct 30 04:51:41.479121 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:41.479055    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-c8tcm\""
Oct 30 04:51:41.479121 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:41.479082    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 04:51:41.547007 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:41.546974    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:51:41.605495 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:41.605463    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/99ed5b75-d71c-496a-a260-794b59289962-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-fgtkv\" (UID: \"99ed5b75-d71c-496a-a260-794b59289962\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fgtkv"
Oct 30 04:51:41.605639 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:41.605514    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqzc8\" (UniqueName: \"kubernetes.io/projected/99ed5b75-d71c-496a-a260-794b59289962-kube-api-access-gqzc8\") pod \"kuadrant-console-plugin-66c4fb575d-fgtkv\" (UID: \"99ed5b75-d71c-496a-a260-794b59289962\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fgtkv"
Oct 30 04:51:41.605639 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:41.605618    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/99ed5b75-d71c-496a-a260-794b59289962-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-fgtkv\" (UID: \"99ed5b75-d71c-496a-a260-794b59289962\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fgtkv"
Oct 30 04:51:41.706250 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:41.706163    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/99ed5b75-d71c-496a-a260-794b59289962-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-fgtkv\" (UID: \"99ed5b75-d71c-496a-a260-794b59289962\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fgtkv"
Oct 30 04:51:41.706250 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:41.706228    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqzc8\" (UniqueName: \"kubernetes.io/projected/99ed5b75-d71c-496a-a260-794b59289962-kube-api-access-gqzc8\") pod \"kuadrant-console-plugin-66c4fb575d-fgtkv\" (UID: \"99ed5b75-d71c-496a-a260-794b59289962\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fgtkv"
Oct 30 04:51:41.706441 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:41.706274    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/99ed5b75-d71c-496a-a260-794b59289962-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-fgtkv\" (UID: \"99ed5b75-d71c-496a-a260-794b59289962\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fgtkv"
Oct 30 04:51:41.706954 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:41.706931    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/99ed5b75-d71c-496a-a260-794b59289962-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-fgtkv\" (UID: \"99ed5b75-d71c-496a-a260-794b59289962\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fgtkv"
Oct 30 04:51:41.708537 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:41.708517    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/99ed5b75-d71c-496a-a260-794b59289962-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-fgtkv\" (UID: \"99ed5b75-d71c-496a-a260-794b59289962\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fgtkv"
Oct 30 04:51:41.724062 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:41.724037    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqzc8\" (UniqueName: \"kubernetes.io/projected/99ed5b75-d71c-496a-a260-794b59289962-kube-api-access-gqzc8\") pod \"kuadrant-console-plugin-66c4fb575d-fgtkv\" (UID: \"99ed5b75-d71c-496a-a260-794b59289962\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fgtkv"
Oct 30 04:51:41.765018 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:41.764982    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fgtkv"
Oct 30 04:51:41.974545 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:41.974466    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:51:42.059542 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:42.059517    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-fgtkv"]
Oct 30 04:51:42.062200 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:51:42.062173    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99ed5b75_d71c_496a_a260_794b59289962.slice/crio-8a8ead4660a1245656984a7dd9d0f0469dae4ea16efbf81f16458e70508afae8 WatchSource:0}: Error finding container 8a8ead4660a1245656984a7dd9d0f0469dae4ea16efbf81f16458e70508afae8: Status 404 returned error can't find the container with id 8a8ead4660a1245656984a7dd9d0f0469dae4ea16efbf81f16458e70508afae8
Oct 30 04:51:42.074139 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:42.074119    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:51:42.632484 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:42.632451    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fgtkv" event={"ID":"99ed5b75-d71c-496a-a260-794b59289962","Type":"ContainerStarted","Data":"8a8ead4660a1245656984a7dd9d0f0469dae4ea16efbf81f16458e70508afae8"}
Oct 30 04:51:46.700498 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:46.700463    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:51:46.800395 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:46.800359    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:51:47.659950 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:47.659911    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fgtkv" event={"ID":"99ed5b75-d71c-496a-a260-794b59289962","Type":"ContainerStarted","Data":"cf73f4987f83785f5fd7a6abaad53e146f64e2ae6da974c9fc6ee0e3e5cb76df"}
Oct 30 04:51:47.696575 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:47.696523    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fgtkv" podStartSLOduration=1.897655822 podStartE2EDuration="6.696509803s" totalImagesPullingTime="4.798853981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:51:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:51:47.695547316 +0000 UTC m=+436.919367248" watchObservedRunningTime="2026-10-30 04:51:47.696509803 +0000 UTC m=+436.920329703"
Oct 30 04:51:50.405284 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:50.405246    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-bjpl8"]
Oct 30 04:51:50.408948 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:50.408896    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-bjpl8"
Oct 30 04:51:50.417009 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:50.416986    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-bjpl8"]
Oct 30 04:51:50.426602 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:50.426580    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-4tst5\""
Oct 30 04:51:50.482507 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:50.482481    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c4abc4da-e543-42bf-8434-65719ffea1b0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-bjpl8\" (UID: \"c4abc4da-e543-42bf-8434-65719ffea1b0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-bjpl8"
Oct 30 04:51:50.482645 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:50.482534    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8x9q\" (UniqueName: \"kubernetes.io/projected/c4abc4da-e543-42bf-8434-65719ffea1b0-kube-api-access-t8x9q\") pod \"kuadrant-operator-controller-manager-6b76784d48-bjpl8\" (UID: \"c4abc4da-e543-42bf-8434-65719ffea1b0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-bjpl8"
Oct 30 04:51:50.583918 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:50.583871    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c4abc4da-e543-42bf-8434-65719ffea1b0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-bjpl8\" (UID: \"c4abc4da-e543-42bf-8434-65719ffea1b0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-bjpl8"
Oct 30 04:51:50.584060 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:50.583975    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8x9q\" (UniqueName: \"kubernetes.io/projected/c4abc4da-e543-42bf-8434-65719ffea1b0-kube-api-access-t8x9q\") pod \"kuadrant-operator-controller-manager-6b76784d48-bjpl8\" (UID: \"c4abc4da-e543-42bf-8434-65719ffea1b0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-bjpl8"
Oct 30 04:51:50.584311 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:50.584290    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c4abc4da-e543-42bf-8434-65719ffea1b0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-bjpl8\" (UID: \"c4abc4da-e543-42bf-8434-65719ffea1b0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-bjpl8"
Oct 30 04:51:50.602652 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:50.602622    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8x9q\" (UniqueName: \"kubernetes.io/projected/c4abc4da-e543-42bf-8434-65719ffea1b0-kube-api-access-t8x9q\") pod \"kuadrant-operator-controller-manager-6b76784d48-bjpl8\" (UID: \"c4abc4da-e543-42bf-8434-65719ffea1b0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-bjpl8"
Oct 30 04:51:50.722328 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:50.722248    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-bjpl8"
Oct 30 04:51:51.039263 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:51.039234    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-bjpl8"]
Oct 30 04:51:51.041741 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:51:51.041717    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4abc4da_e543_42bf_8434_65719ffea1b0.slice/crio-932f69313d42dfe2b372f3816cf75d4c516d8e85b3537e1746168f6467f72479 WatchSource:0}: Error finding container 932f69313d42dfe2b372f3816cf75d4c516d8e85b3537e1746168f6467f72479: Status 404 returned error can't find the container with id 932f69313d42dfe2b372f3816cf75d4c516d8e85b3537e1746168f6467f72479
Oct 30 04:51:51.680653 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:51.680570    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-bjpl8" event={"ID":"c4abc4da-e543-42bf-8434-65719ffea1b0","Type":"ContainerStarted","Data":"932f69313d42dfe2b372f3816cf75d4c516d8e85b3537e1746168f6467f72479"}
Oct 30 04:51:52.767007 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:52.766965    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-bjpl8"]
Oct 30 04:51:52.768814 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:52.768772    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-bjpl8"]
Oct 30 04:51:52.782351 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:52.780716    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-mmjjs"]
Oct 30 04:51:52.792238 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:52.791601    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-mmjjs"
Oct 30 04:51:52.807745 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:52.807721    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-mmjjs"]
Oct 30 04:51:52.903031 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:52.902995    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv6fz\" (UniqueName: \"kubernetes.io/projected/3f7e473e-066e-4bf4-8e59-70da46c26ea2-kube-api-access-xv6fz\") pod \"kuadrant-operator-controller-manager-6b76784d48-mmjjs\" (UID: \"3f7e473e-066e-4bf4-8e59-70da46c26ea2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-mmjjs"
Oct 30 04:51:52.903201 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:52.903045    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f7e473e-066e-4bf4-8e59-70da46c26ea2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-mmjjs\" (UID: \"3f7e473e-066e-4bf4-8e59-70da46c26ea2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-mmjjs"
Oct 30 04:51:53.004168 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:53.004072    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv6fz\" (UniqueName: \"kubernetes.io/projected/3f7e473e-066e-4bf4-8e59-70da46c26ea2-kube-api-access-xv6fz\") pod \"kuadrant-operator-controller-manager-6b76784d48-mmjjs\" (UID: \"3f7e473e-066e-4bf4-8e59-70da46c26ea2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-mmjjs"
Oct 30 04:51:53.004168 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:53.004129    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f7e473e-066e-4bf4-8e59-70da46c26ea2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-mmjjs\" (UID: \"3f7e473e-066e-4bf4-8e59-70da46c26ea2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-mmjjs"
Oct 30 04:51:53.004553 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:53.004534    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f7e473e-066e-4bf4-8e59-70da46c26ea2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-mmjjs\" (UID: \"3f7e473e-066e-4bf4-8e59-70da46c26ea2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-mmjjs"
Oct 30 04:51:53.032304 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:53.032270    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv6fz\" (UniqueName: \"kubernetes.io/projected/3f7e473e-066e-4bf4-8e59-70da46c26ea2-kube-api-access-xv6fz\") pod \"kuadrant-operator-controller-manager-6b76784d48-mmjjs\" (UID: \"3f7e473e-066e-4bf4-8e59-70da46c26ea2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-mmjjs"
Oct 30 04:51:53.115438 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:53.115404    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-mmjjs"
Oct 30 04:51:53.443461 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:53.443429    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-mmjjs"]
Oct 30 04:51:53.446986 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:51:53.446950    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f7e473e_066e_4bf4_8e59_70da46c26ea2.slice/crio-e71b2d0b3b029bef8e12b4b2a5682c16e8d4ebaafdee5270e120bfc6265f2b90 WatchSource:0}: Error finding container e71b2d0b3b029bef8e12b4b2a5682c16e8d4ebaafdee5270e120bfc6265f2b90: Status 404 returned error can't find the container with id e71b2d0b3b029bef8e12b4b2a5682c16e8d4ebaafdee5270e120bfc6265f2b90
Oct 30 04:51:53.692566 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:53.692530    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-mmjjs" event={"ID":"3f7e473e-066e-4bf4-8e59-70da46c26ea2","Type":"ContainerStarted","Data":"e71b2d0b3b029bef8e12b4b2a5682c16e8d4ebaafdee5270e120bfc6265f2b90"}
Oct 30 04:51:54.562098 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:54.562059    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:51:54.658376 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:54.658333    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:51:55.132023 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:55.131981    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:51:55.218870 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:55.218836    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:51:55.704796 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:55.704765    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-bjpl8" podUID="c4abc4da-e543-42bf-8434-65719ffea1b0" containerName="manager" containerID="cri-o://8be751d4cc738b6924c693b94e55d1ed9f325c7d0f8b46faf21fa434d9f63a11" gracePeriod=2
Oct 30 04:51:55.956280 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:55.956254    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-bjpl8"
Oct 30 04:51:55.968930 ip-10-0-135-98 kubenswrapper[5898]: E1030 04:51:55.968884    5898 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b76784d48-bjpl8\" is forbidden: User \"system:node:ip-10-0-135-98.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-98.ec2.internal' and this object" podUID="c4abc4da-e543-42bf-8434-65719ffea1b0" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-bjpl8"
Oct 30 04:51:56.127922 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:56.127870    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4abc4da-e543-42bf-8434-65719ffea1b0-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c4abc4da-e543-42bf-8434-65719ffea1b0-extensions-socket-volume\") pod \"c4abc4da-e543-42bf-8434-65719ffea1b0\" (UID: \"c4abc4da-e543-42bf-8434-65719ffea1b0\") "
Oct 30 04:51:56.128099 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:56.127969    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4abc4da-e543-42bf-8434-65719ffea1b0-kube-api-access-t8x9q\" (UniqueName: \"kubernetes.io/projected/c4abc4da-e543-42bf-8434-65719ffea1b0-kube-api-access-t8x9q\") pod \"c4abc4da-e543-42bf-8434-65719ffea1b0\" (UID: \"c4abc4da-e543-42bf-8434-65719ffea1b0\") "
Oct 30 04:51:56.128171 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:56.128105    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4abc4da-e543-42bf-8434-65719ffea1b0-extensions-socket-volume" pod "c4abc4da-e543-42bf-8434-65719ffea1b0" (UID: "c4abc4da-e543-42bf-8434-65719ffea1b0"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:51:56.130137 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:56.130101    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4abc4da-e543-42bf-8434-65719ffea1b0-kube-api-access-t8x9q" pod "c4abc4da-e543-42bf-8434-65719ffea1b0" (UID: "c4abc4da-e543-42bf-8434-65719ffea1b0"). InnerVolumeSpecName "kube-api-access-t8x9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:51:56.228527 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:56.228460    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8x9q\" (UniqueName: \"kubernetes.io/projected/c4abc4da-e543-42bf-8434-65719ffea1b0-kube-api-access-t8x9q\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 04:51:56.228527 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:56.228485    5898 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c4abc4da-e543-42bf-8434-65719ffea1b0-extensions-socket-volume\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 04:51:56.711061 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:56.711026    5898 generic.go:356] "Generic (PLEG): container finished" podID="c4abc4da-e543-42bf-8434-65719ffea1b0" containerID="8be751d4cc738b6924c693b94e55d1ed9f325c7d0f8b46faf21fa434d9f63a11" exitCode=2
Oct 30 04:51:56.711503 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:56.711074    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-bjpl8"
Oct 30 04:51:56.711503 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:56.711131    5898 scope.go:122] "RemoveContainer" containerID="8be751d4cc738b6924c693b94e55d1ed9f325c7d0f8b46faf21fa434d9f63a11"
Oct 30 04:51:56.713029 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:56.713002    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-mmjjs" event={"ID":"3f7e473e-066e-4bf4-8e59-70da46c26ea2","Type":"ContainerStarted","Data":"a0dbc507769de4a84d3f27ab93dfc67182f2a46418d6270c98d50f63dd036f0d"}
Oct 30 04:51:56.713150 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:56.713131    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-mmjjs"
Oct 30 04:51:56.720353 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:56.720335    5898 scope.go:122] "RemoveContainer" containerID="8be751d4cc738b6924c693b94e55d1ed9f325c7d0f8b46faf21fa434d9f63a11"
Oct 30 04:51:56.720605 ip-10-0-135-98 kubenswrapper[5898]: E1030 04:51:56.720586    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8be751d4cc738b6924c693b94e55d1ed9f325c7d0f8b46faf21fa434d9f63a11\": container with ID starting with 8be751d4cc738b6924c693b94e55d1ed9f325c7d0f8b46faf21fa434d9f63a11 not found: ID does not exist" containerID="8be751d4cc738b6924c693b94e55d1ed9f325c7d0f8b46faf21fa434d9f63a11"
Oct 30 04:51:56.720649 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:56.720612    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8be751d4cc738b6924c693b94e55d1ed9f325c7d0f8b46faf21fa434d9f63a11"} err="failed to get container status \"8be751d4cc738b6924c693b94e55d1ed9f325c7d0f8b46faf21fa434d9f63a11\": rpc error: code = NotFound desc = could not find container \"8be751d4cc738b6924c693b94e55d1ed9f325c7d0f8b46faf21fa434d9f63a11\": container with ID starting with 8be751d4cc738b6924c693b94e55d1ed9f325c7d0f8b46faf21fa434d9f63a11 not found: ID does not exist"
Oct 30 04:51:56.723710 ip-10-0-135-98 kubenswrapper[5898]: E1030 04:51:56.723691    5898 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b76784d48-bjpl8\" is forbidden: User \"system:node:ip-10-0-135-98.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-98.ec2.internal' and this object" podUID="c4abc4da-e543-42bf-8434-65719ffea1b0" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-bjpl8"
Oct 30 04:51:56.736018 ip-10-0-135-98 kubenswrapper[5898]: E1030 04:51:56.735997    5898 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b76784d48-bjpl8\" is forbidden: User \"system:node:ip-10-0-135-98.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-98.ec2.internal' and this object" podUID="c4abc4da-e543-42bf-8434-65719ffea1b0" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-bjpl8"
Oct 30 04:51:56.796693 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:56.796644    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-mmjjs" podStartSLOduration=2.588712949 podStartE2EDuration="4.796629566s" totalImagesPullingTime="2.207916617s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:51:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:51:56.784442579 +0000 UTC m=+446.008262498" watchObservedRunningTime="2026-10-30 04:51:56.796629566 +0000 UTC m=+446.020449466"
Oct 30 04:51:56.817735 ip-10-0-135-98 kubenswrapper[5898]: E1030 04:51:56.817704    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 04:51:56.887681 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:56.887651    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4abc4da-e543-42bf-8434-65719ffea1b0" path="/var/lib/kubelet/pods/c4abc4da-e543-42bf-8434-65719ffea1b0/volumes"
Oct 30 04:51:57.538324 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:57.538294    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:51:57.622022 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:51:57.621984    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:52:00.800803 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:00.800771    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:52:00.884424 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:00.884388    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:52:02.738923 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:02.738872    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:52:02.839310 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:02.839275    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:52:06.495449 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:06.495416    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:52:06.595457 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:06.595415    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:52:07.723928 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:07.723866    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-mmjjs"
Oct 30 04:52:20.412014 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:20.411933    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-mmjjs"]
Oct 30 04:52:20.412536 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:20.412160    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-mmjjs" podUID="3f7e473e-066e-4bf4-8e59-70da46c26ea2" containerName="manager" containerID="cri-o://a0dbc507769de4a84d3f27ab93dfc67182f2a46418d6270c98d50f63dd036f0d" gracePeriod=10
Oct 30 04:52:20.665285 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:20.665232    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-mmjjs"
Oct 30 04:52:20.723998 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:20.723970    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f7e473e-066e-4bf4-8e59-70da46c26ea2-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f7e473e-066e-4bf4-8e59-70da46c26ea2-extensions-socket-volume\") pod \"3f7e473e-066e-4bf4-8e59-70da46c26ea2\" (UID: \"3f7e473e-066e-4bf4-8e59-70da46c26ea2\") "
Oct 30 04:52:20.724121 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:20.724026    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f7e473e-066e-4bf4-8e59-70da46c26ea2-kube-api-access-xv6fz\" (UniqueName: \"kubernetes.io/projected/3f7e473e-066e-4bf4-8e59-70da46c26ea2-kube-api-access-xv6fz\") pod \"3f7e473e-066e-4bf4-8e59-70da46c26ea2\" (UID: \"3f7e473e-066e-4bf4-8e59-70da46c26ea2\") "
Oct 30 04:52:20.724574 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:20.724539    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f7e473e-066e-4bf4-8e59-70da46c26ea2-extensions-socket-volume" pod "3f7e473e-066e-4bf4-8e59-70da46c26ea2" (UID: "3f7e473e-066e-4bf4-8e59-70da46c26ea2"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:52:20.725997 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:20.725970    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f7e473e-066e-4bf4-8e59-70da46c26ea2-kube-api-access-xv6fz" pod "3f7e473e-066e-4bf4-8e59-70da46c26ea2" (UID: "3f7e473e-066e-4bf4-8e59-70da46c26ea2"). InnerVolumeSpecName "kube-api-access-xv6fz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:52:20.824512 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:20.824484    5898 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f7e473e-066e-4bf4-8e59-70da46c26ea2-extensions-socket-volume\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 04:52:20.824512 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:20.824508    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv6fz\" (UniqueName: \"kubernetes.io/projected/3f7e473e-066e-4bf4-8e59-70da46c26ea2-kube-api-access-xv6fz\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 04:52:20.826199 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:20.826174    5898 generic.go:356] "Generic (PLEG): container finished" podID="3f7e473e-066e-4bf4-8e59-70da46c26ea2" containerID="a0dbc507769de4a84d3f27ab93dfc67182f2a46418d6270c98d50f63dd036f0d" exitCode=0
Oct 30 04:52:20.826299 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:20.826219    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-mmjjs"
Oct 30 04:52:20.826338 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:20.826219    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-mmjjs" event={"ID":"3f7e473e-066e-4bf4-8e59-70da46c26ea2","Type":"ContainerDied","Data":"a0dbc507769de4a84d3f27ab93dfc67182f2a46418d6270c98d50f63dd036f0d"}
Oct 30 04:52:20.826376 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:20.826349    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-mmjjs" event={"ID":"3f7e473e-066e-4bf4-8e59-70da46c26ea2","Type":"ContainerDied","Data":"e71b2d0b3b029bef8e12b4b2a5682c16e8d4ebaafdee5270e120bfc6265f2b90"}
Oct 30 04:52:20.826409 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:20.826376    5898 scope.go:122] "RemoveContainer" containerID="a0dbc507769de4a84d3f27ab93dfc67182f2a46418d6270c98d50f63dd036f0d"
Oct 30 04:52:20.835695 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:20.835677    5898 scope.go:122] "RemoveContainer" containerID="a0dbc507769de4a84d3f27ab93dfc67182f2a46418d6270c98d50f63dd036f0d"
Oct 30 04:52:20.835989 ip-10-0-135-98 kubenswrapper[5898]: E1030 04:52:20.835968    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0dbc507769de4a84d3f27ab93dfc67182f2a46418d6270c98d50f63dd036f0d\": container with ID starting with a0dbc507769de4a84d3f27ab93dfc67182f2a46418d6270c98d50f63dd036f0d not found: ID does not exist" containerID="a0dbc507769de4a84d3f27ab93dfc67182f2a46418d6270c98d50f63dd036f0d"
Oct 30 04:52:20.836061 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:20.836002    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0dbc507769de4a84d3f27ab93dfc67182f2a46418d6270c98d50f63dd036f0d"} err="failed to get container status \"a0dbc507769de4a84d3f27ab93dfc67182f2a46418d6270c98d50f63dd036f0d\": rpc error: code = NotFound desc = could not find container \"a0dbc507769de4a84d3f27ab93dfc67182f2a46418d6270c98d50f63dd036f0d\": container with ID starting with a0dbc507769de4a84d3f27ab93dfc67182f2a46418d6270c98d50f63dd036f0d not found: ID does not exist"
Oct 30 04:52:20.873386 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:20.873357    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-mmjjs"]
Oct 30 04:52:20.889203 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:20.889176    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-mmjjs"]
Oct 30 04:52:22.887974 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:22.887938    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f7e473e-066e-4bf4-8e59-70da46c26ea2" path="/var/lib/kubelet/pods/3f7e473e-066e-4bf4-8e59-70da46c26ea2/volumes"
Oct 30 04:52:25.297505 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:25.297473    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5"]
Oct 30 04:52:25.297876 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:25.297807    5898 cpu_manager.go:414] "containerMap: removing container" podUID="c4abc4da-e543-42bf-8434-65719ffea1b0" containerName="manager"
Oct 30 04:52:25.297876 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:25.297818    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4abc4da-e543-42bf-8434-65719ffea1b0" containerName="manager"
Oct 30 04:52:25.297876 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:25.297837    5898 cpu_manager.go:414] "containerMap: removing container" podUID="3f7e473e-066e-4bf4-8e59-70da46c26ea2" containerName="manager"
Oct 30 04:52:25.297876 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:25.297842    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f7e473e-066e-4bf4-8e59-70da46c26ea2" containerName="manager"
Oct 30 04:52:25.298050 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:25.297917    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f7e473e-066e-4bf4-8e59-70da46c26ea2" containerName="manager"
Oct 30 04:52:25.298050 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:25.297933    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4abc4da-e543-42bf-8434-65719ffea1b0" containerName="manager"
Oct 30 04:52:25.306047 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:25.306023    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5"
Oct 30 04:52:25.319682 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:25.319655    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5"]
Oct 30 04:52:25.328980 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:25.328958    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-dr8cs\""
Oct 30 04:52:25.343951 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:25.343923    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:52:25.364914 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:25.364883    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b79b4fcb-1af2-424b-9b56-f6f9d40f88b2-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5\" (UID: \"b79b4fcb-1af2-424b-9b56-f6f9d40f88b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5"
Oct 30 04:52:25.365047 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:25.364932    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b79b4fcb-1af2-424b-9b56-f6f9d40f88b2-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5\" (UID: \"b79b4fcb-1af2-424b-9b56-f6f9d40f88b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5"
Oct 30 04:52:25.365047 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:25.364951    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktsqj\" (UniqueName: \"kubernetes.io/projected/b79b4fcb-1af2-424b-9b56-f6f9d40f88b2-kube-api-access-ktsqj\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5\" (UID: \"b79b4fcb-1af2-424b-9b56-f6f9d40f88b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5"
Oct 30 04:52:25.365126 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:25.365097    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b79b4fcb-1af2-424b-9b56-f6f9d40f88b2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5\" (UID: \"b79b4fcb-1af2-424b-9b56-f6f9d40f88b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5"
Oct 30 04:52:25.365172 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:25.365131    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b79b4fcb-1af2-424b-9b56-f6f9d40f88b2-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5\" (UID: \"b79b4fcb-1af2-424b-9b56-f6f9d40f88b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5"
Oct 30 04:52:25.365172 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:25.365153    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b79b4fcb-1af2-424b-9b56-f6f9d40f88b2-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5\" (UID: \"b79b4fcb-1af2-424b-9b56-f6f9d40f88b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5"
Oct 30 04:52:25.365254 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:25.365175    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b79b4fcb-1af2-424b-9b56-f6f9d40f88b2-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5\" (UID: \"b79b4fcb-1af2-424b-9b56-f6f9d40f88b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5"
Oct 30 04:52:25.365254 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:25.365232    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b79b4fcb-1af2-424b-9b56-f6f9d40f88b2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5\" (UID: \"b79b4fcb-1af2-424b-9b56-f6f9d40f88b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5"
Oct 30 04:52:25.365345 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:25.365331    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b79b4fcb-1af2-424b-9b56-f6f9d40f88b2-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5\" (UID: \"b79b4fcb-1af2-424b-9b56-f6f9d40f88b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5"
Oct 30 04:52:25.444114 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:25.444080    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:52:25.466289 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:25.466263    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b79b4fcb-1af2-424b-9b56-f6f9d40f88b2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5\" (UID: \"b79b4fcb-1af2-424b-9b56-f6f9d40f88b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5"
Oct 30 04:52:25.466400 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:25.466308    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b79b4fcb-1af2-424b-9b56-f6f9d40f88b2-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5\" (UID: \"b79b4fcb-1af2-424b-9b56-f6f9d40f88b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5"
Oct 30 04:52:25.466400 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:25.466342    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b79b4fcb-1af2-424b-9b56-f6f9d40f88b2-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5\" (UID: \"b79b4fcb-1af2-424b-9b56-f6f9d40f88b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5"
Oct 30 04:52:25.466508 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:25.466468    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b79b4fcb-1af2-424b-9b56-f6f9d40f88b2-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5\" (UID: \"b79b4fcb-1af2-424b-9b56-f6f9d40f88b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5"
Oct 30 04:52:25.466563 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:25.466512    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktsqj\" (UniqueName: \"kubernetes.io/projected/b79b4fcb-1af2-424b-9b56-f6f9d40f88b2-kube-api-access-ktsqj\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5\" (UID: \"b79b4fcb-1af2-424b-9b56-f6f9d40f88b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5"
Oct 30 04:52:25.466617 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:25.466572    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b79b4fcb-1af2-424b-9b56-f6f9d40f88b2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5\" (UID: \"b79b4fcb-1af2-424b-9b56-f6f9d40f88b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5"
Oct 30 04:52:25.466617 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:25.466611    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b79b4fcb-1af2-424b-9b56-f6f9d40f88b2-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5\" (UID: \"b79b4fcb-1af2-424b-9b56-f6f9d40f88b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5"
Oct 30 04:52:25.466715 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:25.466636    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b79b4fcb-1af2-424b-9b56-f6f9d40f88b2-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5\" (UID: \"b79b4fcb-1af2-424b-9b56-f6f9d40f88b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5"
Oct 30 04:52:25.466715 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:25.466668    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b79b4fcb-1af2-424b-9b56-f6f9d40f88b2-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5\" (UID: \"b79b4fcb-1af2-424b-9b56-f6f9d40f88b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5"
Oct 30 04:52:25.466817 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:25.466721    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b79b4fcb-1af2-424b-9b56-f6f9d40f88b2-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5\" (UID: \"b79b4fcb-1af2-424b-9b56-f6f9d40f88b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5"
Oct 30 04:52:25.466817 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:25.466752    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b79b4fcb-1af2-424b-9b56-f6f9d40f88b2-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5\" (UID: \"b79b4fcb-1af2-424b-9b56-f6f9d40f88b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5"
Oct 30 04:52:25.467039 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:25.467015    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b79b4fcb-1af2-424b-9b56-f6f9d40f88b2-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5\" (UID: \"b79b4fcb-1af2-424b-9b56-f6f9d40f88b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5"
Oct 30 04:52:25.467109 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:25.467045    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b79b4fcb-1af2-424b-9b56-f6f9d40f88b2-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5\" (UID: \"b79b4fcb-1af2-424b-9b56-f6f9d40f88b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5"
Oct 30 04:52:25.467258 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:25.467243    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b79b4fcb-1af2-424b-9b56-f6f9d40f88b2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5\" (UID: \"b79b4fcb-1af2-424b-9b56-f6f9d40f88b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5"
Oct 30 04:52:25.468985 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:25.468958    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b79b4fcb-1af2-424b-9b56-f6f9d40f88b2-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5\" (UID: \"b79b4fcb-1af2-424b-9b56-f6f9d40f88b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5"
Oct 30 04:52:25.469174 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:25.469152    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b79b4fcb-1af2-424b-9b56-f6f9d40f88b2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5\" (UID: \"b79b4fcb-1af2-424b-9b56-f6f9d40f88b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5"
Oct 30 04:52:25.484739 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:25.484715    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktsqj\" (UniqueName: \"kubernetes.io/projected/b79b4fcb-1af2-424b-9b56-f6f9d40f88b2-kube-api-access-ktsqj\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5\" (UID: \"b79b4fcb-1af2-424b-9b56-f6f9d40f88b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5"
Oct 30 04:52:25.485163 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:25.485141    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b79b4fcb-1af2-424b-9b56-f6f9d40f88b2-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5\" (UID: \"b79b4fcb-1af2-424b-9b56-f6f9d40f88b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5"
Oct 30 04:52:25.619521 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:25.619488    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5"
Oct 30 04:52:25.953431 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:25.953351    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5"]
Oct 30 04:52:25.956190 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:52:25.956159    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb79b4fcb_1af2_424b_9b56_f6f9d40f88b2.slice/crio-3246751a7c268101c7d3576ce6199d884e0ec4d3f4f6f6172b42506dfc1bfb1d WatchSource:0}: Error finding container 3246751a7c268101c7d3576ce6199d884e0ec4d3f4f6f6172b42506dfc1bfb1d: Status 404 returned error can't find the container with id 3246751a7c268101c7d3576ce6199d884e0ec4d3f4f6f6172b42506dfc1bfb1d
Oct 30 04:52:25.958018 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:25.957988    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 04:52:25.958087 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:25.958049    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 04:52:25.958087 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:25.958077    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 04:52:26.817230 ip-10-0-135-98 kubenswrapper[5898]: E1030 04:52:26.817192    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 04:52:26.857170 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:26.857136    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5" event={"ID":"b79b4fcb-1af2-424b-9b56-f6f9d40f88b2","Type":"ContainerStarted","Data":"a3b5c5c8abf9c7aa2bc2f46518da6052e93a0a128479282af76d080aad3be592"}
Oct 30 04:52:26.857312 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:26.857184    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5" event={"ID":"b79b4fcb-1af2-424b-9b56-f6f9d40f88b2","Type":"ContainerStarted","Data":"3246751a7c268101c7d3576ce6199d884e0ec4d3f4f6f6172b42506dfc1bfb1d"}
Oct 30 04:52:26.918237 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:26.918191    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5" podStartSLOduration=1.918177468 podStartE2EDuration="1.918177468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:52:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:52:26.906584204 +0000 UTC m=+476.130404106" watchObservedRunningTime="2026-10-30 04:52:26.918177468 +0000 UTC m=+476.141997369"
Oct 30 04:52:27.619889 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:27.619856    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5"
Oct 30 04:52:27.624318 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:27.624295    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5"
Oct 30 04:52:27.861952 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:27.861922    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5"
Oct 30 04:52:27.862869 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:27.862851    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5"
Oct 30 04:52:37.413507 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:37.413474    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:52:40.552955 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:40.552894    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:52:40.671961 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:40.671932    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:52:40.771058 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:40.771021    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:52:41.478315 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:41.478268    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 04:52:41.481626 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:41.481604    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-n8njr"
Oct 30 04:52:41.497525 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:41.497503    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 30 04:52:41.499330 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:41.499310    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 04:52:41.503863 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:41.503840    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:52:41.557430 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:41.557403    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 04:52:41.584859 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:41.584838    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:52:41.588066 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:41.588043    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd76n\" (UniqueName: \"kubernetes.io/projected/50ad4653-5ecb-4f30-9533-2ba45e1fa1cd-kube-api-access-jd76n\") pod \"limitador-limitador-7c9b87d574-n8njr\" (UID: \"50ad4653-5ecb-4f30-9533-2ba45e1fa1cd\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-n8njr"
Oct 30 04:52:41.588154 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:41.588075    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/50ad4653-5ecb-4f30-9533-2ba45e1fa1cd-config-file\") pod \"limitador-limitador-7c9b87d574-n8njr\" (UID: \"50ad4653-5ecb-4f30-9533-2ba45e1fa1cd\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-n8njr"
Oct 30 04:52:41.689570 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:41.689543    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd76n\" (UniqueName: \"kubernetes.io/projected/50ad4653-5ecb-4f30-9533-2ba45e1fa1cd-kube-api-access-jd76n\") pod \"limitador-limitador-7c9b87d574-n8njr\" (UID: \"50ad4653-5ecb-4f30-9533-2ba45e1fa1cd\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-n8njr"
Oct 30 04:52:41.689570 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:41.689574    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/50ad4653-5ecb-4f30-9533-2ba45e1fa1cd-config-file\") pod \"limitador-limitador-7c9b87d574-n8njr\" (UID: \"50ad4653-5ecb-4f30-9533-2ba45e1fa1cd\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-n8njr"
Oct 30 04:52:41.690244 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:41.690227    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/50ad4653-5ecb-4f30-9533-2ba45e1fa1cd-config-file\") pod \"limitador-limitador-7c9b87d574-n8njr\" (UID: \"50ad4653-5ecb-4f30-9533-2ba45e1fa1cd\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-n8njr"
Oct 30 04:52:41.707481 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:41.707453    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd76n\" (UniqueName: \"kubernetes.io/projected/50ad4653-5ecb-4f30-9533-2ba45e1fa1cd-kube-api-access-jd76n\") pod \"limitador-limitador-7c9b87d574-n8njr\" (UID: \"50ad4653-5ecb-4f30-9533-2ba45e1fa1cd\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-n8njr"
Oct 30 04:52:41.793661 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:41.793592    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-n8njr"
Oct 30 04:52:42.084743 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:42.084715    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 04:52:42.086550 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:52:42.086522    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50ad4653_5ecb_4f30_9533_2ba45e1fa1cd.slice/crio-3ee52e3fc3f40f1c3e8b1683a79587c6ecf42d746b185935ec119c7beb79e745 WatchSource:0}: Error finding container 3ee52e3fc3f40f1c3e8b1683a79587c6ecf42d746b185935ec119c7beb79e745: Status 404 returned error can't find the container with id 3ee52e3fc3f40f1c3e8b1683a79587c6ecf42d746b185935ec119c7beb79e745
Oct 30 04:52:42.929579 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:42.929543    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-n8njr" event={"ID":"50ad4653-5ecb-4f30-9533-2ba45e1fa1cd","Type":"ContainerStarted","Data":"3ee52e3fc3f40f1c3e8b1683a79587c6ecf42d746b185935ec119c7beb79e745"}
Oct 30 04:52:43.935186 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:43.935149    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-n8njr" event={"ID":"50ad4653-5ecb-4f30-9533-2ba45e1fa1cd","Type":"ContainerStarted","Data":"ec3f87af98414bfb1157b8518489bfc8f84c4a6f3be3b700430734533896d3ea"}
Oct 30 04:52:43.935663 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:43.935267    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-n8njr"
Oct 30 04:52:43.972332 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:43.972286    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-n8njr" podStartSLOduration=2.072492229 podStartE2EDuration="2.972273224s" totalImagesPullingTime="899.780995ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:52:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:52:43.971206829 +0000 UTC m=+493.195026729" watchObservedRunningTime="2026-10-30 04:52:43.972273224 +0000 UTC m=+493.196093125"
Oct 30 04:52:54.940602 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:52:54.940570    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-n8njr"
Oct 30 04:52:56.817195 ip-10-0-135-98 kubenswrapper[5898]: E1030 04:52:56.817157    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 04:53:00.988772 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:53:00.988740    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:53:01.108164 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:53:01.108131    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:53:01.720252 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:53:01.720219    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:53:14.192454 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:53:14.192422    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:53:14.927548 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:53:14.927519    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:53:16.515815 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:53:16.515772    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:53:23.608755 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:53:23.608721    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:53:23.695614 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:53:23.695581    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:53:23.776764 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:53:23.776728    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:53:23.906730 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:53:23.906651    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:53:24.007444 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:53:24.007410    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:53:24.959443 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:53:24.959414    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:53:25.033174 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:53:25.033142    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:53:26.816377 ip-10-0-135-98 kubenswrapper[5898]: E1030 04:53:26.816342    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 04:53:27.288344 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:53:27.288315    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:53:27.387234 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:53:27.387204    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:53:30.932974 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:53:30.932944    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:53:31.035595 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:53:31.035567    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:53:36.291874 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:53:36.291835    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:53:37.521662 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:53:37.521625    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:53:44.667965 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:53:44.667870    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:53:44.740881 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:53:44.740848    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:53:56.816848 ip-10-0-135-98 kubenswrapper[5898]: E1030 04:53:56.816815    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 04:53:59.654915 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:53:59.654878    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:53:59.812330 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:53:59.812297    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:54:01.125306 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:01.125268    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:54:01.225889 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:01.225854    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:54:01.326501 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:01.326474    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:54:01.426198 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:01.426103    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:54:03.222733 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:03.222690    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 04:54:12.071139 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:12.071102    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-9jkvx"]
Oct 30 04:54:12.075668 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:12.075650    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-9jkvx"
Oct 30 04:54:12.091060 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:12.091031    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-9jkvx"]
Oct 30 04:54:12.239618 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:12.239590    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-966zl\" (UniqueName: \"kubernetes.io/projected/62287fbd-b4d1-4b24-9e5f-25aff2c09664-kube-api-access-966zl\") pod \"llmisvc-controller-manager-6466c554d8-9jkvx\" (UID: \"62287fbd-b4d1-4b24-9e5f-25aff2c09664\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-9jkvx"
Oct 30 04:54:12.239773 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:12.239708    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/62287fbd-b4d1-4b24-9e5f-25aff2c09664-cert\") pod \"llmisvc-controller-manager-6466c554d8-9jkvx\" (UID: \"62287fbd-b4d1-4b24-9e5f-25aff2c09664\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-9jkvx"
Oct 30 04:54:12.340578 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:12.340547    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/62287fbd-b4d1-4b24-9e5f-25aff2c09664-cert\") pod \"llmisvc-controller-manager-6466c554d8-9jkvx\" (UID: \"62287fbd-b4d1-4b24-9e5f-25aff2c09664\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-9jkvx"
Oct 30 04:54:12.340727 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:12.340593    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-966zl\" (UniqueName: \"kubernetes.io/projected/62287fbd-b4d1-4b24-9e5f-25aff2c09664-kube-api-access-966zl\") pod \"llmisvc-controller-manager-6466c554d8-9jkvx\" (UID: \"62287fbd-b4d1-4b24-9e5f-25aff2c09664\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-9jkvx"
Oct 30 04:54:12.343025 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:12.343000    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/62287fbd-b4d1-4b24-9e5f-25aff2c09664-cert\") pod \"llmisvc-controller-manager-6466c554d8-9jkvx\" (UID: \"62287fbd-b4d1-4b24-9e5f-25aff2c09664\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-9jkvx"
Oct 30 04:54:12.344691 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:12.344668    5898 generic.go:356] "Generic (PLEG): container finished" podID="b79b4fcb-1af2-424b-9b56-f6f9d40f88b2" containerID="a3b5c5c8abf9c7aa2bc2f46518da6052e93a0a128479282af76d080aad3be592" exitCode=0
Oct 30 04:54:12.344808 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:12.344706    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5" event={"ID":"b79b4fcb-1af2-424b-9b56-f6f9d40f88b2","Type":"ContainerDied","Data":"a3b5c5c8abf9c7aa2bc2f46518da6052e93a0a128479282af76d080aad3be592"}
Oct 30 04:54:12.345072 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:12.345053    5898 scope.go:122] "RemoveContainer" containerID="a3b5c5c8abf9c7aa2bc2f46518da6052e93a0a128479282af76d080aad3be592"
Oct 30 04:54:12.346038 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:12.346013    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 04:54:12.346115 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:12.346078    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 04:54:12.346115 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:12.346110    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 04:54:12.360054 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:12.360019    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-966zl\" (UniqueName: \"kubernetes.io/projected/62287fbd-b4d1-4b24-9e5f-25aff2c09664-kube-api-access-966zl\") pod \"llmisvc-controller-manager-6466c554d8-9jkvx\" (UID: \"62287fbd-b4d1-4b24-9e5f-25aff2c09664\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-9jkvx"
Oct 30 04:54:12.389702 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:12.389678    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-9jkvx"
Oct 30 04:54:12.703235 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:12.703151    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-9jkvx"]
Oct 30 04:54:12.706083 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:54:12.706051    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62287fbd_b4d1_4b24_9e5f_25aff2c09664.slice/crio-a33c4d493ed682d29e44d1b8bda1dd6efd0052b2fc1eb538cdacb8c696a84735 WatchSource:0}: Error finding container a33c4d493ed682d29e44d1b8bda1dd6efd0052b2fc1eb538cdacb8c696a84735: Status 404 returned error can't find the container with id a33c4d493ed682d29e44d1b8bda1dd6efd0052b2fc1eb538cdacb8c696a84735
Oct 30 04:54:12.707634 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:12.707616    5898 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 04:54:12.862741 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:12.862704    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5"
Oct 30 04:54:13.350679 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:13.350641    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5" event={"ID":"b79b4fcb-1af2-424b-9b56-f6f9d40f88b2","Type":"ContainerStarted","Data":"282e46ca48092a154a2b59493f81130381368d68255ab2da8caf4eb46868e91f"}
Oct 30 04:54:13.352144 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:13.352119    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-9jkvx" event={"ID":"62287fbd-b4d1-4b24-9e5f-25aff2c09664","Type":"ContainerStarted","Data":"542d972471eed4c60c78d76c4ac18666ffce50481ed09423f674e98236f5d7ed"}
Oct 30 04:54:13.352264 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:13.352157    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-9jkvx" event={"ID":"62287fbd-b4d1-4b24-9e5f-25aff2c09664","Type":"ContainerStarted","Data":"a33c4d493ed682d29e44d1b8bda1dd6efd0052b2fc1eb538cdacb8c696a84735"}
Oct 30 04:54:13.352264 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:13.352203    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-9jkvx"
Oct 30 04:54:13.445453 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:13.445362    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-9jkvx" podStartSLOduration=0.983216532 podStartE2EDuration="1.445349177s" totalImagesPullingTime="462.132645ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:54:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:54:13.444047233 +0000 UTC m=+582.667867134" watchObservedRunningTime="2026-10-30 04:54:13.445349177 +0000 UTC m=+582.669169077"
Oct 30 04:54:13.619856 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:13.619825    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5"
Oct 30 04:54:13.621020 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:13.620990    5898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 04:54:13.621146 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:13.621037    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5" podUID="b79b4fcb-1af2-424b-9b56-f6f9d40f88b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 04:54:14.620380 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:14.620346    5898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 04:54:14.620792 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:14.620410    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5" podUID="b79b4fcb-1af2-424b-9b56-f6f9d40f88b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 04:54:15.620094 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:15.620060    5898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 04:54:15.620263 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:15.620110    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5" podUID="b79b4fcb-1af2-424b-9b56-f6f9d40f88b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 04:54:16.620499 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:16.620455    5898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 04:54:16.621022 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:16.620518    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5" podUID="b79b4fcb-1af2-424b-9b56-f6f9d40f88b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 04:54:17.620719 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:17.620676    5898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 04:54:17.621238 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:17.620735    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5" podUID="b79b4fcb-1af2-424b-9b56-f6f9d40f88b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 04:54:18.624095 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:18.624065    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5"
Oct 30 04:54:18.624499 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:18.624374    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5"
Oct 30 04:54:18.625104 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:18.625087    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5"
Oct 30 04:54:26.816864 ip-10-0-135-98 kubenswrapper[5898]: E1030 04:54:26.816827    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 04:54:36.337968 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:36.337930    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:54:36.415947 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:36.415893    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:54:37.653975 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:37.653930    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:54:44.358804 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:44.358776    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-9jkvx"
Oct 30 04:54:44.420319 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:44.420283    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-57mpg"]
Oct 30 04:54:44.420527 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:44.420493    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-57mpg" podUID="0a25543c-936a-4521-b874-2ae3f9dcc61c" containerName="manager" containerID="cri-o://91fa7371a9a1c85d4ff068605ac5613d4b298e8fe6ed02a0ec676ad8f5bd9637" gracePeriod=30
Oct 30 04:54:44.663435 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:44.663411    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-57mpg"
Oct 30 04:54:44.720211 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:44.720183    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a25543c-936a-4521-b874-2ae3f9dcc61c-cert\" (UniqueName: \"kubernetes.io/secret/0a25543c-936a-4521-b874-2ae3f9dcc61c-cert\") pod \"0a25543c-936a-4521-b874-2ae3f9dcc61c\" (UID: \"0a25543c-936a-4521-b874-2ae3f9dcc61c\") "
Oct 30 04:54:44.720367 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:44.720220    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a25543c-936a-4521-b874-2ae3f9dcc61c-kube-api-access-dj825\" (UniqueName: \"kubernetes.io/projected/0a25543c-936a-4521-b874-2ae3f9dcc61c-kube-api-access-dj825\") pod \"0a25543c-936a-4521-b874-2ae3f9dcc61c\" (UID: \"0a25543c-936a-4521-b874-2ae3f9dcc61c\") "
Oct 30 04:54:44.722369 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:44.722336    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a25543c-936a-4521-b874-2ae3f9dcc61c-cert" pod "0a25543c-936a-4521-b874-2ae3f9dcc61c" (UID: "0a25543c-936a-4521-b874-2ae3f9dcc61c"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:54:44.722474 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:44.722389    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a25543c-936a-4521-b874-2ae3f9dcc61c-kube-api-access-dj825" pod "0a25543c-936a-4521-b874-2ae3f9dcc61c" (UID: "0a25543c-936a-4521-b874-2ae3f9dcc61c"). InnerVolumeSpecName "kube-api-access-dj825". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:54:44.820779 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:44.820743    5898 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0a25543c-936a-4521-b874-2ae3f9dcc61c-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 04:54:44.820779 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:44.820777    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dj825\" (UniqueName: \"kubernetes.io/projected/0a25543c-936a-4521-b874-2ae3f9dcc61c-kube-api-access-dj825\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 04:54:45.222014 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:45.221964    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:54:45.323777 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:45.323729    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:54:45.495371 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:45.495286    5898 generic.go:356] "Generic (PLEG): container finished" podID="0a25543c-936a-4521-b874-2ae3f9dcc61c" containerID="91fa7371a9a1c85d4ff068605ac5613d4b298e8fe6ed02a0ec676ad8f5bd9637" exitCode=0
Oct 30 04:54:45.495371 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:45.495328    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-57mpg"
Oct 30 04:54:45.495856 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:45.495367    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-57mpg" event={"ID":"0a25543c-936a-4521-b874-2ae3f9dcc61c","Type":"ContainerDied","Data":"91fa7371a9a1c85d4ff068605ac5613d4b298e8fe6ed02a0ec676ad8f5bd9637"}
Oct 30 04:54:45.495856 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:45.495418    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-57mpg" event={"ID":"0a25543c-936a-4521-b874-2ae3f9dcc61c","Type":"ContainerDied","Data":"73b0d96609f504ac148e37c11d83ebe812cfda0d7d8045e4bdd3bf9598d2c9f5"}
Oct 30 04:54:45.495856 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:45.495440    5898 scope.go:122] "RemoveContainer" containerID="91fa7371a9a1c85d4ff068605ac5613d4b298e8fe6ed02a0ec676ad8f5bd9637"
Oct 30 04:54:45.503921 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:45.503869    5898 scope.go:122] "RemoveContainer" containerID="91fa7371a9a1c85d4ff068605ac5613d4b298e8fe6ed02a0ec676ad8f5bd9637"
Oct 30 04:54:45.504210 ip-10-0-135-98 kubenswrapper[5898]: E1030 04:54:45.504191    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91fa7371a9a1c85d4ff068605ac5613d4b298e8fe6ed02a0ec676ad8f5bd9637\": container with ID starting with 91fa7371a9a1c85d4ff068605ac5613d4b298e8fe6ed02a0ec676ad8f5bd9637 not found: ID does not exist" containerID="91fa7371a9a1c85d4ff068605ac5613d4b298e8fe6ed02a0ec676ad8f5bd9637"
Oct 30 04:54:45.504260 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:45.504219    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91fa7371a9a1c85d4ff068605ac5613d4b298e8fe6ed02a0ec676ad8f5bd9637"} err="failed to get container status \"91fa7371a9a1c85d4ff068605ac5613d4b298e8fe6ed02a0ec676ad8f5bd9637\": rpc error: code = NotFound desc = could not find container \"91fa7371a9a1c85d4ff068605ac5613d4b298e8fe6ed02a0ec676ad8f5bd9637\": container with ID starting with 91fa7371a9a1c85d4ff068605ac5613d4b298e8fe6ed02a0ec676ad8f5bd9637 not found: ID does not exist"
Oct 30 04:54:45.546482 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:45.546455    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-57mpg"]
Oct 30 04:54:45.558054 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:45.558028    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-57mpg"]
Oct 30 04:54:46.888033 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:46.888001    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a25543c-936a-4521-b874-2ae3f9dcc61c" path="/var/lib/kubelet/pods/0a25543c-936a-4521-b874-2ae3f9dcc61c/volumes"
Oct 30 04:54:56.388329 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:54:56.388294    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:54:56.816823 ip-10-0-135-98 kubenswrapper[5898]: E1030 04:54:56.816747    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 04:55:02.637299 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:02.637247    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:55:03.711877 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:03.711842    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 04:55:04.873236 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:04.873203    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:55:06.393754 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:06.393712    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:55:07.892483 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:07.892453    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 04:55:08.891578 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:08.891548    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:55:10.065073 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:10.065038    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:55:11.888125 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:11.888090    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:55:12.938692 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:12.938658    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:55:13.193351 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:13.193274    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 04:55:19.214009 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:19.213970    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:55:19.314236 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:19.314200    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:55:19.483590 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:19.483509    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 04:55:26.818259 ip-10-0-135-98 kubenswrapper[5898]: E1030 04:55:26.818222    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 04:55:27.778085 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:27.778046    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:55:27.944516 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:27.944479    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:55:33.400562 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:33.400526    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:55:33.495123 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:33.495090    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:55:37.755422 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:37.755394    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:55:55.894586 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:55.894551    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89"]
Oct 30 04:55:55.895037 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:55.894951    5898 cpu_manager.go:414] "containerMap: removing container" podUID="0a25543c-936a-4521-b874-2ae3f9dcc61c" containerName="manager"
Oct 30 04:55:55.895037 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:55.894965    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a25543c-936a-4521-b874-2ae3f9dcc61c" containerName="manager"
Oct 30 04:55:55.895037 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:55.895030    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a25543c-936a-4521-b874-2ae3f9dcc61c" containerName="manager"
Oct 30 04:55:55.898026 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:55.898009    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89"
Oct 30 04:55:55.917619 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:55.917596    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89"]
Oct 30 04:55:56.014921 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:56.014877    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01b63947-ab2c-4052-91a9-f91a1a499168-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-ngz89\" (UID: \"01b63947-ab2c-4052-91a9-f91a1a499168\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89"
Oct 30 04:55:56.015085 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:56.014934    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01b63947-ab2c-4052-91a9-f91a1a499168-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-ngz89\" (UID: \"01b63947-ab2c-4052-91a9-f91a1a499168\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89"
Oct 30 04:55:56.015202 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:56.015181    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01b63947-ab2c-4052-91a9-f91a1a499168-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-ngz89\" (UID: \"01b63947-ab2c-4052-91a9-f91a1a499168\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89"
Oct 30 04:55:56.015272 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:56.015213    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01b63947-ab2c-4052-91a9-f91a1a499168-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-ngz89\" (UID: \"01b63947-ab2c-4052-91a9-f91a1a499168\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89"
Oct 30 04:55:56.015480 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:56.015460    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01b63947-ab2c-4052-91a9-f91a1a499168-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-ngz89\" (UID: \"01b63947-ab2c-4052-91a9-f91a1a499168\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89"
Oct 30 04:55:56.015583 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:56.015559    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01b63947-ab2c-4052-91a9-f91a1a499168-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-ngz89\" (UID: \"01b63947-ab2c-4052-91a9-f91a1a499168\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89"
Oct 30 04:55:56.015687 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:56.015669    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nstq6\" (UniqueName: \"kubernetes.io/projected/01b63947-ab2c-4052-91a9-f91a1a499168-kube-api-access-nstq6\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-ngz89\" (UID: \"01b63947-ab2c-4052-91a9-f91a1a499168\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89"
Oct 30 04:55:56.015774 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:56.015758    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01b63947-ab2c-4052-91a9-f91a1a499168-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-ngz89\" (UID: \"01b63947-ab2c-4052-91a9-f91a1a499168\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89"
Oct 30 04:55:56.015810 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:56.015791    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01b63947-ab2c-4052-91a9-f91a1a499168-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-ngz89\" (UID: \"01b63947-ab2c-4052-91a9-f91a1a499168\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89"
Oct 30 04:55:56.116606 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:56.116577    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01b63947-ab2c-4052-91a9-f91a1a499168-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-ngz89\" (UID: \"01b63947-ab2c-4052-91a9-f91a1a499168\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89"
Oct 30 04:55:56.116606 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:56.116606    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01b63947-ab2c-4052-91a9-f91a1a499168-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-ngz89\" (UID: \"01b63947-ab2c-4052-91a9-f91a1a499168\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89"
Oct 30 04:55:56.116778 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:56.116622    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01b63947-ab2c-4052-91a9-f91a1a499168-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-ngz89\" (UID: \"01b63947-ab2c-4052-91a9-f91a1a499168\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89"
Oct 30 04:55:56.116778 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:56.116637    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01b63947-ab2c-4052-91a9-f91a1a499168-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-ngz89\" (UID: \"01b63947-ab2c-4052-91a9-f91a1a499168\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89"
Oct 30 04:55:56.116778 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:56.116680    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01b63947-ab2c-4052-91a9-f91a1a499168-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-ngz89\" (UID: \"01b63947-ab2c-4052-91a9-f91a1a499168\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89"
Oct 30 04:55:56.116778 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:56.116717    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01b63947-ab2c-4052-91a9-f91a1a499168-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-ngz89\" (UID: \"01b63947-ab2c-4052-91a9-f91a1a499168\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89"
Oct 30 04:55:56.116778 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:56.116742    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nstq6\" (UniqueName: \"kubernetes.io/projected/01b63947-ab2c-4052-91a9-f91a1a499168-kube-api-access-nstq6\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-ngz89\" (UID: \"01b63947-ab2c-4052-91a9-f91a1a499168\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89"
Oct 30 04:55:56.116778 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:56.116768    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01b63947-ab2c-4052-91a9-f91a1a499168-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-ngz89\" (UID: \"01b63947-ab2c-4052-91a9-f91a1a499168\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89"
Oct 30 04:55:56.117142 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:56.116791    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01b63947-ab2c-4052-91a9-f91a1a499168-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-ngz89\" (UID: \"01b63947-ab2c-4052-91a9-f91a1a499168\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89"
Oct 30 04:55:56.117142 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:56.117049    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01b63947-ab2c-4052-91a9-f91a1a499168-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-ngz89\" (UID: \"01b63947-ab2c-4052-91a9-f91a1a499168\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89"
Oct 30 04:55:56.117142 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:56.117074    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01b63947-ab2c-4052-91a9-f91a1a499168-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-ngz89\" (UID: \"01b63947-ab2c-4052-91a9-f91a1a499168\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89"
Oct 30 04:55:56.117291 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:56.117270    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01b63947-ab2c-4052-91a9-f91a1a499168-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-ngz89\" (UID: \"01b63947-ab2c-4052-91a9-f91a1a499168\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89"
Oct 30 04:55:56.117338 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:56.117309    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01b63947-ab2c-4052-91a9-f91a1a499168-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-ngz89\" (UID: \"01b63947-ab2c-4052-91a9-f91a1a499168\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89"
Oct 30 04:55:56.117513 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:56.117492    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01b63947-ab2c-4052-91a9-f91a1a499168-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-ngz89\" (UID: \"01b63947-ab2c-4052-91a9-f91a1a499168\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89"
Oct 30 04:55:56.119120 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:56.119098    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01b63947-ab2c-4052-91a9-f91a1a499168-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-ngz89\" (UID: \"01b63947-ab2c-4052-91a9-f91a1a499168\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89"
Oct 30 04:55:56.119320 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:56.119300    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01b63947-ab2c-4052-91a9-f91a1a499168-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-ngz89\" (UID: \"01b63947-ab2c-4052-91a9-f91a1a499168\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89"
Oct 30 04:55:56.134367 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:56.134341    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01b63947-ab2c-4052-91a9-f91a1a499168-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-ngz89\" (UID: \"01b63947-ab2c-4052-91a9-f91a1a499168\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89"
Oct 30 04:55:56.134452 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:56.134404    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nstq6\" (UniqueName: \"kubernetes.io/projected/01b63947-ab2c-4052-91a9-f91a1a499168-kube-api-access-nstq6\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-ngz89\" (UID: \"01b63947-ab2c-4052-91a9-f91a1a499168\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89"
Oct 30 04:55:56.210450 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:56.210392    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89"
Oct 30 04:55:56.513678 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:56.513654    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89"]
Oct 30 04:55:56.515741 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:55:56.515713    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01b63947_ab2c_4052_91a9_f91a1a499168.slice/crio-cb6cbd44d7abbd46568a93e64deffe13fd39e886c2ce3d23df4ac3322256ffe7 WatchSource:0}: Error finding container cb6cbd44d7abbd46568a93e64deffe13fd39e886c2ce3d23df4ac3322256ffe7: Status 404 returned error can't find the container with id cb6cbd44d7abbd46568a93e64deffe13fd39e886c2ce3d23df4ac3322256ffe7
Oct 30 04:55:56.517831 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:56.517796    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 04:55:56.517930 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:56.517867    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 04:55:56.517930 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:56.517922    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 04:55:56.819492 ip-10-0-135-98 kubenswrapper[5898]: E1030 04:55:56.819412    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 04:55:56.837489 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:56.837453    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89" event={"ID":"01b63947-ab2c-4052-91a9-f91a1a499168","Type":"ContainerStarted","Data":"b27bd1182ec2712d85a1e46a896fc322e493c44280df6cda6df4c6a59fab14d4"}
Oct 30 04:55:56.837644 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:56.837521    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89" event={"ID":"01b63947-ab2c-4052-91a9-f91a1a499168","Type":"ContainerStarted","Data":"cb6cbd44d7abbd46568a93e64deffe13fd39e886c2ce3d23df4ac3322256ffe7"}
Oct 30 04:55:56.876553 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:56.876496    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89" podStartSLOduration=1.876465108 podStartE2EDuration="1.876465108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:55:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:55:56.875793636 +0000 UTC m=+686.099613555" watchObservedRunningTime="2026-10-30 04:55:56.876465108 +0000 UTC m=+686.100285010"
Oct 30 04:55:57.210866 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:57.210833    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89"
Oct 30 04:55:57.212080 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:57.212054    5898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Oct 30 04:55:57.212204 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:57.212112    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89" podUID="01b63947-ab2c-4052-91a9-f91a1a499168" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Oct 30 04:55:58.157578 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:58.157546    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:55:58.211795 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:58.211754    5898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Oct 30 04:55:58.212190 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:58.211812    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89" podUID="01b63947-ab2c-4052-91a9-f91a1a499168" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Oct 30 04:55:58.665076 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:58.665040    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:55:59.211099 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:59.211064    5898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Oct 30 04:55:59.211272 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:55:59.211117    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89" podUID="01b63947-ab2c-4052-91a9-f91a1a499168" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Oct 30 04:56:00.211348 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:00.211310    5898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Oct 30 04:56:00.211766 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:00.211366    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89" podUID="01b63947-ab2c-4052-91a9-f91a1a499168" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Oct 30 04:56:00.346310 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:00.346279    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:56:00.451064 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:00.451035    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:56:01.211473 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:01.211434    5898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Oct 30 04:56:01.211862 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:01.211507    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89" podUID="01b63947-ab2c-4052-91a9-f91a1a499168" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Oct 30 04:56:01.893924 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:01.891287    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:56:01.998927 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:01.996300    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:56:02.211592 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:02.211508    5898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Oct 30 04:56:02.211592 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:02.211558    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89" podUID="01b63947-ab2c-4052-91a9-f91a1a499168" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Oct 30 04:56:03.210783 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:03.210743    5898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Oct 30 04:56:03.211008 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:03.210818    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89" podUID="01b63947-ab2c-4052-91a9-f91a1a499168" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Oct 30 04:56:04.754926 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:04.754876    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89"
Oct 30 04:56:04.755412 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:04.755286    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89"
Oct 30 04:56:04.756131 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:04.756108    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89"
Oct 30 04:56:26.817303 ip-10-0-135-98 kubenswrapper[5898]: E1030 04:56:26.817266    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 04:56:35.383282 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:35.383223    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:56:35.485079 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:35.485049    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:56:35.632498 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:35.632471    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:56:35.922577 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:35.922513    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:56:36.047359 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:36.047323    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:56:36.147525 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:36.147492    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:56:36.253260 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:36.253169    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:56:36.352948 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:36.352909    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:56:36.450368 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:36.450333    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:56:36.552810 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:36.552737    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:56:36.658809 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:36.658776    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:56:36.752245 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:36.752205    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:56:36.904210 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:36.904182    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:56:37.871253 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:37.871216    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:56:39.363799 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:39.363765    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:56:39.464162 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:39.464132    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:56:40.363360 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:40.363326    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:56:40.466712 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:40.466675    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:56:41.072910 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:41.072878    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:56:41.194170 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:41.194139    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:56:41.318183 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:41.318153    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:56:41.504514 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:41.504431    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:56:41.610433 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:41.610403    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:56:41.926412 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:41.926385    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:56:42.028884 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:42.028814    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:56:42.178453 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:42.178380    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:56:42.303739 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:42.303706    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:56:42.989619 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:42.989587    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:56:43.093789 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:43.090224    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:56:43.193302 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:43.193267    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:56:43.392531 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:43.392493    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:56:44.920139 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:44.920109    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:56:45.001046 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:45.001009    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:56:45.101376 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:45.101342    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:56:45.521952 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:45.521883    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:56:45.614034 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:45.613982    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:56:46.351460 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:46.351429    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:56:46.438645 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:46.438590    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:56:46.624810 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:46.624733    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 04:56:46.704753 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:46.704719    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:56:46.804816 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:46.804789    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:56:47.028099 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:47.028012    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:56:47.919876 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:47.919842    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:56:48.011916 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:48.011865    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:56:48.035357 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:48.035333    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b21507-59b47dc888-29s5k"]
Oct 30 04:56:48.038726 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:48.038706    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-b21507-59b47dc888-29s5k"
Oct 30 04:56:48.066295 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:48.066259    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-b21507\""
Oct 30 04:56:48.066295 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:48.066276    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-b21507-dockercfg-gdvxh\""
Oct 30 04:56:48.067827 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:48.067808    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b21507-59b47dc888-29s5k"]
Oct 30 04:56:48.095520 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:48.095470    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b21507-b7c9cc4c5-bkrt5"]
Oct 30 04:56:48.098804 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:48.098789    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b21507-b7c9cc4c5-bkrt5"
Oct 30 04:56:48.116082 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:48.116052    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b21507-b7c9cc4c5-bkrt5"]
Oct 30 04:56:48.164097 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:48.164070    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30460201-04af-49fb-a7d3-5ee121e3bdb4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-b21507-59b47dc888-29s5k\" (UID: \"30460201-04af-49fb-a7d3-5ee121e3bdb4\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b21507-59b47dc888-29s5k"
Oct 30 04:56:48.164242 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:48.164129    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mznvr\" (UniqueName: \"kubernetes.io/projected/30460201-04af-49fb-a7d3-5ee121e3bdb4-kube-api-access-mznvr\") pod \"payload-processing-e2e-worker-w0-b21507-59b47dc888-29s5k\" (UID: \"30460201-04af-49fb-a7d3-5ee121e3bdb4\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b21507-59b47dc888-29s5k"
Oct 30 04:56:48.164242 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:48.164181    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5zlr\" (UniqueName: \"kubernetes.io/projected/c00385be-b173-422b-8ba6-0985c9cf7598-kube-api-access-v5zlr\") pod \"payload-pre-processing-e2e-worker-w0-b21507-b7c9cc4c5-bkrt5\" (UID: \"c00385be-b173-422b-8ba6-0985c9cf7598\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b21507-b7c9cc4c5-bkrt5"
Oct 30 04:56:48.164324 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:48.164281    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c00385be-b173-422b-8ba6-0985c9cf7598-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-b21507-b7c9cc4c5-bkrt5\" (UID: \"c00385be-b173-422b-8ba6-0985c9cf7598\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b21507-b7c9cc4c5-bkrt5"
Oct 30 04:56:48.265497 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:48.265395    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mznvr\" (UniqueName: \"kubernetes.io/projected/30460201-04af-49fb-a7d3-5ee121e3bdb4-kube-api-access-mznvr\") pod \"payload-processing-e2e-worker-w0-b21507-59b47dc888-29s5k\" (UID: \"30460201-04af-49fb-a7d3-5ee121e3bdb4\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b21507-59b47dc888-29s5k"
Oct 30 04:56:48.265497 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:48.265477    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5zlr\" (UniqueName: \"kubernetes.io/projected/c00385be-b173-422b-8ba6-0985c9cf7598-kube-api-access-v5zlr\") pod \"payload-pre-processing-e2e-worker-w0-b21507-b7c9cc4c5-bkrt5\" (UID: \"c00385be-b173-422b-8ba6-0985c9cf7598\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b21507-b7c9cc4c5-bkrt5"
Oct 30 04:56:48.265727 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:48.265578    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c00385be-b173-422b-8ba6-0985c9cf7598-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-b21507-b7c9cc4c5-bkrt5\" (UID: \"c00385be-b173-422b-8ba6-0985c9cf7598\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b21507-b7c9cc4c5-bkrt5"
Oct 30 04:56:48.265727 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:48.265610    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30460201-04af-49fb-a7d3-5ee121e3bdb4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-b21507-59b47dc888-29s5k\" (UID: \"30460201-04af-49fb-a7d3-5ee121e3bdb4\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b21507-59b47dc888-29s5k"
Oct 30 04:56:48.266662 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:48.266635    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c00385be-b173-422b-8ba6-0985c9cf7598-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-b21507-b7c9cc4c5-bkrt5\" (UID: \"c00385be-b173-422b-8ba6-0985c9cf7598\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b21507-b7c9cc4c5-bkrt5"
Oct 30 04:56:48.266796 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:48.266775    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30460201-04af-49fb-a7d3-5ee121e3bdb4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-b21507-59b47dc888-29s5k\" (UID: \"30460201-04af-49fb-a7d3-5ee121e3bdb4\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b21507-59b47dc888-29s5k"
Oct 30 04:56:48.283770 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:48.283742    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mznvr\" (UniqueName: \"kubernetes.io/projected/30460201-04af-49fb-a7d3-5ee121e3bdb4-kube-api-access-mznvr\") pod \"payload-processing-e2e-worker-w0-b21507-59b47dc888-29s5k\" (UID: \"30460201-04af-49fb-a7d3-5ee121e3bdb4\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b21507-59b47dc888-29s5k"
Oct 30 04:56:48.283770 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:48.283763    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5zlr\" (UniqueName: \"kubernetes.io/projected/c00385be-b173-422b-8ba6-0985c9cf7598-kube-api-access-v5zlr\") pod \"payload-pre-processing-e2e-worker-w0-b21507-b7c9cc4c5-bkrt5\" (UID: \"c00385be-b173-422b-8ba6-0985c9cf7598\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b21507-b7c9cc4c5-bkrt5"
Oct 30 04:56:48.350989 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:48.350959    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-b21507-59b47dc888-29s5k"
Oct 30 04:56:48.411678 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:48.411496    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b21507-b7c9cc4c5-bkrt5"
Oct 30 04:56:48.644235 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:48.644203    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b21507-59b47dc888-29s5k"]
Oct 30 04:56:48.647565 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:56:48.647539    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30460201_04af_49fb_a7d3_5ee121e3bdb4.slice/crio-547b195778b359aafee4133965662df0392a839ab42592baaa1d40e70ed49172 WatchSource:0}: Error finding container 547b195778b359aafee4133965662df0392a839ab42592baaa1d40e70ed49172: Status 404 returned error can't find the container with id 547b195778b359aafee4133965662df0392a839ab42592baaa1d40e70ed49172
Oct 30 04:56:48.663167 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:48.663145    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:56:48.707476 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:48.707442    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b21507-b7c9cc4c5-bkrt5"]
Oct 30 04:56:48.709294 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:56:48.709271    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc00385be_b173_422b_8ba6_0985c9cf7598.slice/crio-896b5d7594672fd4df164a73bcfafd39a7fdcee6f159f7dedbcf1d5ed85c5c30 WatchSource:0}: Error finding container 896b5d7594672fd4df164a73bcfafd39a7fdcee6f159f7dedbcf1d5ed85c5c30: Status 404 returned error can't find the container with id 896b5d7594672fd4df164a73bcfafd39a7fdcee6f159f7dedbcf1d5ed85c5c30
Oct 30 04:56:48.842851 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:48.842815    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:56:48.940741 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:48.940666    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:56:49.105975 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:49.105671    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-b21507-59b47dc888-29s5k" event={"ID":"30460201-04af-49fb-a7d3-5ee121e3bdb4","Type":"ContainerStarted","Data":"547b195778b359aafee4133965662df0392a839ab42592baaa1d40e70ed49172"}
Oct 30 04:56:49.113753 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:49.113710    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b21507-b7c9cc4c5-bkrt5" event={"ID":"c00385be-b173-422b-8ba6-0985c9cf7598","Type":"ContainerStarted","Data":"896b5d7594672fd4df164a73bcfafd39a7fdcee6f159f7dedbcf1d5ed85c5c30"}
Oct 30 04:56:50.425930 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:50.425843    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:56:50.507719 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:50.507679    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:56:50.598584 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:50.598551    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:56:50.846467 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:50.846444    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:56:51.126665 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:51.126624    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b21507-b7c9cc4c5-bkrt5" event={"ID":"c00385be-b173-422b-8ba6-0985c9cf7598","Type":"ContainerStarted","Data":"b37375a2fb7eaaf4020e964ad8c46fb6f2cee83c7c4ab28a874c230648b93568"}
Oct 30 04:56:51.126861 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:51.126694    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b21507-b7c9cc4c5-bkrt5"
Oct 30 04:56:51.128199 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:51.128175    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-b21507-59b47dc888-29s5k" event={"ID":"30460201-04af-49fb-a7d3-5ee121e3bdb4","Type":"ContainerStarted","Data":"4ba9604e0c21f1b6d817fe4ca8f0099336f3de54d1b09c6a9ac0170423555310"}
Oct 30 04:56:51.128313 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:51.128302    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-b21507-59b47dc888-29s5k"
Oct 30 04:56:51.187447 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:51.187399    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b21507-b7c9cc4c5-bkrt5" podStartSLOduration=1.128487775 podStartE2EDuration="3.187384504s" totalImagesPullingTime="2.058896729s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:56:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:56:51.185662462 +0000 UTC m=+740.409482364" watchObservedRunningTime="2026-10-30 04:56:51.187384504 +0000 UTC m=+740.411204426"
Oct 30 04:56:51.249645 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:51.249588    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-b21507-59b47dc888-29s5k" podStartSLOduration=1.133426505 podStartE2EDuration="3.249573805s" totalImagesPullingTime="2.1161473s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:56:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:56:51.248107852 +0000 UTC m=+740.471927766" watchObservedRunningTime="2026-10-30 04:56:51.249573805 +0000 UTC m=+740.473393708"
Oct 30 04:56:51.268704 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:51.268672    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 04:56:52.486402 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:52.486375    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:56:52.573536 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:52.573484    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:56:52.910121 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:52.910063    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:56:54.346753 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:54.346717    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:56:54.446707 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:54.446673    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:56:54.558386 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:54.558358    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:56:56.067695 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:56.067658    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-889ca9-c6b65f8fc-7jqcd"]
Oct 30 04:56:56.071382 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:56.071365    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-889ca9-c6b65f8fc-7jqcd"
Oct 30 04:56:56.099940 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:56.099912    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-889ca9-serving-cert\""
Oct 30 04:56:56.102046 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:56.102022    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-889ca9-c6b65f8fc-7jqcd"]
Oct 30 04:56:56.102580 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:56.102563    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 04:56:56.102679 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:56.102620    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 04:56:56.102779 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:56.102764    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-grc2l\""
Oct 30 04:56:56.102840 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:56.102816    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 04:56:56.154445 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:56.154415    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:56:56.233472 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:56.233435    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:56:56.243545 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:56.243523    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llbvb\" (UniqueName: \"kubernetes.io/projected/5d3b67cf-5656-4d62-8adc-91df8cdcf394-kube-api-access-llbvb\") pod \"maas-api-e2e-worker-w3-889ca9-c6b65f8fc-7jqcd\" (UID: \"5d3b67cf-5656-4d62-8adc-91df8cdcf394\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-889ca9-c6b65f8fc-7jqcd"
Oct 30 04:56:56.243626 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:56.243582    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d3b67cf-5656-4d62-8adc-91df8cdcf394-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-889ca9-c6b65f8fc-7jqcd\" (UID: \"5d3b67cf-5656-4d62-8adc-91df8cdcf394\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-889ca9-c6b65f8fc-7jqcd"
Oct 30 04:56:56.243736 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:56.243718    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d3b67cf-5656-4d62-8adc-91df8cdcf394-metrics-certs\") pod \"maas-api-e2e-worker-w3-889ca9-c6b65f8fc-7jqcd\" (UID: \"5d3b67cf-5656-4d62-8adc-91df8cdcf394\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-889ca9-c6b65f8fc-7jqcd"
Oct 30 04:56:56.243802 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:56.243780    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d3b67cf-5656-4d62-8adc-91df8cdcf394-maas-api-tls\") pod \"maas-api-e2e-worker-w3-889ca9-c6b65f8fc-7jqcd\" (UID: \"5d3b67cf-5656-4d62-8adc-91df8cdcf394\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-889ca9-c6b65f8fc-7jqcd"
Oct 30 04:56:56.344924 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:56.344871    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d3b67cf-5656-4d62-8adc-91df8cdcf394-maas-api-tls\") pod \"maas-api-e2e-worker-w3-889ca9-c6b65f8fc-7jqcd\" (UID: \"5d3b67cf-5656-4d62-8adc-91df8cdcf394\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-889ca9-c6b65f8fc-7jqcd"
Oct 30 04:56:56.344924 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:56.344919    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llbvb\" (UniqueName: \"kubernetes.io/projected/5d3b67cf-5656-4d62-8adc-91df8cdcf394-kube-api-access-llbvb\") pod \"maas-api-e2e-worker-w3-889ca9-c6b65f8fc-7jqcd\" (UID: \"5d3b67cf-5656-4d62-8adc-91df8cdcf394\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-889ca9-c6b65f8fc-7jqcd"
Oct 30 04:56:56.345162 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:56.344972    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d3b67cf-5656-4d62-8adc-91df8cdcf394-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-889ca9-c6b65f8fc-7jqcd\" (UID: \"5d3b67cf-5656-4d62-8adc-91df8cdcf394\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-889ca9-c6b65f8fc-7jqcd"
Oct 30 04:56:56.345162 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:56.345009    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d3b67cf-5656-4d62-8adc-91df8cdcf394-metrics-certs\") pod \"maas-api-e2e-worker-w3-889ca9-c6b65f8fc-7jqcd\" (UID: \"5d3b67cf-5656-4d62-8adc-91df8cdcf394\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-889ca9-c6b65f8fc-7jqcd"
Oct 30 04:56:56.345162 ip-10-0-135-98 kubenswrapper[5898]: E1030 04:56:56.345031    5898 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-889ca9-serving-cert: secret "maas-api-e2e-worker-w3-889ca9-serving-cert" not found
Oct 30 04:56:56.345162 ip-10-0-135-98 kubenswrapper[5898]: E1030 04:56:56.345109    5898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d3b67cf-5656-4d62-8adc-91df8cdcf394-maas-api-tls podName:5d3b67cf-5656-4d62-8adc-91df8cdcf394 nodeName:}" failed. No retries permitted until 2026-10-30 04:56:56.845094489 +0000 UTC m=+746.068914386 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5d3b67cf-5656-4d62-8adc-91df8cdcf394-maas-api-tls") pod "maas-api-e2e-worker-w3-889ca9-c6b65f8fc-7jqcd" (UID: "5d3b67cf-5656-4d62-8adc-91df8cdcf394") : secret "maas-api-e2e-worker-w3-889ca9-serving-cert" not found
Oct 30 04:56:56.345694 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:56.345677    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d3b67cf-5656-4d62-8adc-91df8cdcf394-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-889ca9-c6b65f8fc-7jqcd\" (UID: \"5d3b67cf-5656-4d62-8adc-91df8cdcf394\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-889ca9-c6b65f8fc-7jqcd"
Oct 30 04:56:56.347264 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:56.347248    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d3b67cf-5656-4d62-8adc-91df8cdcf394-metrics-certs\") pod \"maas-api-e2e-worker-w3-889ca9-c6b65f8fc-7jqcd\" (UID: \"5d3b67cf-5656-4d62-8adc-91df8cdcf394\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-889ca9-c6b65f8fc-7jqcd"
Oct 30 04:56:56.363530 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:56.363507    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llbvb\" (UniqueName: \"kubernetes.io/projected/5d3b67cf-5656-4d62-8adc-91df8cdcf394-kube-api-access-llbvb\") pod \"maas-api-e2e-worker-w3-889ca9-c6b65f8fc-7jqcd\" (UID: \"5d3b67cf-5656-4d62-8adc-91df8cdcf394\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-889ca9-c6b65f8fc-7jqcd"
Oct 30 04:56:56.453401 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:56.453374    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:56:56.613332 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:56.613256    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 04:56:56.816592 ip-10-0-135-98 kubenswrapper[5898]: E1030 04:56:56.816550    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 04:56:56.848336 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:56.848309    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d3b67cf-5656-4d62-8adc-91df8cdcf394-maas-api-tls\") pod \"maas-api-e2e-worker-w3-889ca9-c6b65f8fc-7jqcd\" (UID: \"5d3b67cf-5656-4d62-8adc-91df8cdcf394\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-889ca9-c6b65f8fc-7jqcd"
Oct 30 04:56:56.850517 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:56.850496    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d3b67cf-5656-4d62-8adc-91df8cdcf394-maas-api-tls\") pod \"maas-api-e2e-worker-w3-889ca9-c6b65f8fc-7jqcd\" (UID: \"5d3b67cf-5656-4d62-8adc-91df8cdcf394\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-889ca9-c6b65f8fc-7jqcd"
Oct 30 04:56:56.983888 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:56.983809    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-889ca9-c6b65f8fc-7jqcd"
Oct 30 04:56:57.283334 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:57.283308    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-889ca9-c6b65f8fc-7jqcd"]
Oct 30 04:56:57.285969 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:56:57.285940    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d3b67cf_5656_4d62_8adc_91df8cdcf394.slice/crio-cabce4aeefa8bc80fdf33b52e59c6f7a8f7c4112874b6bdc698119e6c09f272c WatchSource:0}: Error finding container cabce4aeefa8bc80fdf33b52e59c6f7a8f7c4112874b6bdc698119e6c09f272c: Status 404 returned error can't find the container with id cabce4aeefa8bc80fdf33b52e59c6f7a8f7c4112874b6bdc698119e6c09f272c
Oct 30 04:56:58.061869 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:58.061839    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:56:58.164064 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:56:58.163986    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-889ca9-c6b65f8fc-7jqcd" event={"ID":"5d3b67cf-5656-4d62-8adc-91df8cdcf394","Type":"ContainerStarted","Data":"cabce4aeefa8bc80fdf33b52e59c6f7a8f7c4112874b6bdc698119e6c09f272c"}
Oct 30 04:57:00.027133 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:00.027090    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:57:00.175575 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:00.175537    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-889ca9-c6b65f8fc-7jqcd" event={"ID":"5d3b67cf-5656-4d62-8adc-91df8cdcf394","Type":"ContainerStarted","Data":"3153ab5f05a819303fe834d19e1e494243595628b4cf0191e9c78aec14b3628e"}
Oct 30 04:57:00.175779 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:00.175636    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-889ca9-c6b65f8fc-7jqcd"
Oct 30 04:57:00.214117 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:00.214042    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-889ca9-c6b65f8fc-7jqcd" podStartSLOduration=2.008018021 podStartE2EDuration="4.214020031s" totalImagesPullingTime="2.20600201s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:56:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:57:00.212974146 +0000 UTC m=+749.436794075" watchObservedRunningTime="2026-10-30 04:57:00.214020031 +0000 UTC m=+749.437839935"
Oct 30 04:57:01.935367 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:01.935329    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 04:57:02.043698 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:02.043670    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:57:02.136748 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:02.136721    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-b21507-59b47dc888-29s5k"
Oct 30 04:57:02.136961 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:02.136798    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b21507-b7c9cc4c5-bkrt5"
Oct 30 04:57:04.134630 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:04.134599    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:57:04.536123 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:04.536043    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:57:05.186099 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:05.186060    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:57:05.392695 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:05.392653    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:57:05.492140 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:05.491993    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:57:05.662629 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:05.662593    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:57:05.763145 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:05.763047    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:57:06.185733 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:06.185679    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-889ca9-c6b65f8fc-7jqcd"
Oct 30 04:57:06.293829 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:06.293800    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:57:06.497701 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:06.497625    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:57:06.598205 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:06.598176    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:57:08.398171 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:08.398138    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:57:09.787749 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:09.787707    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49"]
Oct 30 04:57:09.791631 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:09.791611    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49"
Oct 30 04:57:09.813571 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:09.813536    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49"]
Oct 30 04:57:09.819822 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:09.819795    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-397179\"/\"openshift-service-ca.crt\""
Oct 30 04:57:09.823627 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:09.823586    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-397179\"/\"default-dockercfg-lmv86\""
Oct 30 04:57:09.823752 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:09.823670    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-397179\"/\"simulator-w2-397179-kserve-self-signed-certs\""
Oct 30 04:57:09.823811 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:09.823776    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-397179\"/\"kube-root-ca.crt\""
Oct 30 04:57:09.848642 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:09.848610    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:57:09.850979 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:09.850961    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05eef4eb-f379-4dc7-a947-123bd85a4968-tmp-dir\") pod \"simulator-w2-397179-kserve-57b67456f7-n4g49\" (UID: \"05eef4eb-f379-4dc7-a947-123bd85a4968\") " pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49"
Oct 30 04:57:09.851038 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:09.850988    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05eef4eb-f379-4dc7-a947-123bd85a4968-tls-certs\") pod \"simulator-w2-397179-kserve-57b67456f7-n4g49\" (UID: \"05eef4eb-f379-4dc7-a947-123bd85a4968\") " pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49"
Oct 30 04:57:09.851038 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:09.851009    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05eef4eb-f379-4dc7-a947-123bd85a4968-dshm\") pod \"simulator-w2-397179-kserve-57b67456f7-n4g49\" (UID: \"05eef4eb-f379-4dc7-a947-123bd85a4968\") " pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49"
Oct 30 04:57:09.851148 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:09.851136    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05eef4eb-f379-4dc7-a947-123bd85a4968-model-cache\") pod \"simulator-w2-397179-kserve-57b67456f7-n4g49\" (UID: \"05eef4eb-f379-4dc7-a947-123bd85a4968\") " pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49"
Oct 30 04:57:09.851209 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:09.851197    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtpk5\" (UniqueName: \"kubernetes.io/projected/05eef4eb-f379-4dc7-a947-123bd85a4968-kube-api-access-rtpk5\") pod \"simulator-w2-397179-kserve-57b67456f7-n4g49\" (UID: \"05eef4eb-f379-4dc7-a947-123bd85a4968\") " pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49"
Oct 30 04:57:09.851246 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:09.851217    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05eef4eb-f379-4dc7-a947-123bd85a4968-home\") pod \"simulator-w2-397179-kserve-57b67456f7-n4g49\" (UID: \"05eef4eb-f379-4dc7-a947-123bd85a4968\") " pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49"
Oct 30 04:57:09.951795 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:09.951757    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05eef4eb-f379-4dc7-a947-123bd85a4968-dshm\") pod \"simulator-w2-397179-kserve-57b67456f7-n4g49\" (UID: \"05eef4eb-f379-4dc7-a947-123bd85a4968\") " pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49"
Oct 30 04:57:09.952010 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:09.951837    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05eef4eb-f379-4dc7-a947-123bd85a4968-model-cache\") pod \"simulator-w2-397179-kserve-57b67456f7-n4g49\" (UID: \"05eef4eb-f379-4dc7-a947-123bd85a4968\") " pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49"
Oct 30 04:57:09.952010 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:09.951921    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtpk5\" (UniqueName: \"kubernetes.io/projected/05eef4eb-f379-4dc7-a947-123bd85a4968-kube-api-access-rtpk5\") pod \"simulator-w2-397179-kserve-57b67456f7-n4g49\" (UID: \"05eef4eb-f379-4dc7-a947-123bd85a4968\") " pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49"
Oct 30 04:57:09.952010 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:09.951948    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05eef4eb-f379-4dc7-a947-123bd85a4968-home\") pod \"simulator-w2-397179-kserve-57b67456f7-n4g49\" (UID: \"05eef4eb-f379-4dc7-a947-123bd85a4968\") " pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49"
Oct 30 04:57:09.952010 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:09.951997    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05eef4eb-f379-4dc7-a947-123bd85a4968-tmp-dir\") pod \"simulator-w2-397179-kserve-57b67456f7-n4g49\" (UID: \"05eef4eb-f379-4dc7-a947-123bd85a4968\") " pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49"
Oct 30 04:57:09.952247 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:09.952020    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05eef4eb-f379-4dc7-a947-123bd85a4968-tls-certs\") pod \"simulator-w2-397179-kserve-57b67456f7-n4g49\" (UID: \"05eef4eb-f379-4dc7-a947-123bd85a4968\") " pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49"
Oct 30 04:57:09.952332 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:09.952299    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05eef4eb-f379-4dc7-a947-123bd85a4968-model-cache\") pod \"simulator-w2-397179-kserve-57b67456f7-n4g49\" (UID: \"05eef4eb-f379-4dc7-a947-123bd85a4968\") " pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49"
Oct 30 04:57:09.952445 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:09.952361    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05eef4eb-f379-4dc7-a947-123bd85a4968-home\") pod \"simulator-w2-397179-kserve-57b67456f7-n4g49\" (UID: \"05eef4eb-f379-4dc7-a947-123bd85a4968\") " pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49"
Oct 30 04:57:09.952502 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:09.952454    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05eef4eb-f379-4dc7-a947-123bd85a4968-tmp-dir\") pod \"simulator-w2-397179-kserve-57b67456f7-n4g49\" (UID: \"05eef4eb-f379-4dc7-a947-123bd85a4968\") " pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49"
Oct 30 04:57:09.954148 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:09.954128    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05eef4eb-f379-4dc7-a947-123bd85a4968-dshm\") pod \"simulator-w2-397179-kserve-57b67456f7-n4g49\" (UID: \"05eef4eb-f379-4dc7-a947-123bd85a4968\") " pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49"
Oct 30 04:57:09.954583 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:09.954557    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05eef4eb-f379-4dc7-a947-123bd85a4968-tls-certs\") pod \"simulator-w2-397179-kserve-57b67456f7-n4g49\" (UID: \"05eef4eb-f379-4dc7-a947-123bd85a4968\") " pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49"
Oct 30 04:57:09.971340 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:09.971313    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtpk5\" (UniqueName: \"kubernetes.io/projected/05eef4eb-f379-4dc7-a947-123bd85a4968-kube-api-access-rtpk5\") pod \"simulator-w2-397179-kserve-57b67456f7-n4g49\" (UID: \"05eef4eb-f379-4dc7-a947-123bd85a4968\") " pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49"
Oct 30 04:57:10.106082 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:10.106036    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49"
Oct 30 04:57:10.401310 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:10.401134    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49"]
Oct 30 04:57:10.403478 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:57:10.403452    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod05eef4eb_f379_4dc7_a947_123bd85a4968.slice/crio-b9d069cd11fe41b79c6ed98ce4ad656ef542f11338edd54ac01c42e5f06d177a WatchSource:0}: Error finding container b9d069cd11fe41b79c6ed98ce4ad656ef542f11338edd54ac01c42e5f06d177a: Status 404 returned error can't find the container with id b9d069cd11fe41b79c6ed98ce4ad656ef542f11338edd54ac01c42e5f06d177a
Oct 30 04:57:10.650730 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:10.650647    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:57:10.811795 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:10.811760    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:57:11.233209 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:11.233168    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49" event={"ID":"05eef4eb-f379-4dc7-a947-123bd85a4968","Type":"ContainerStarted","Data":"b9d069cd11fe41b79c6ed98ce4ad656ef542f11338edd54ac01c42e5f06d177a"}
Oct 30 04:57:11.447098 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:11.447023    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:57:11.549521 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:11.549498    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:57:11.646317 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:11.646282    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:57:12.239233 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:12.239197    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49" event={"ID":"05eef4eb-f379-4dc7-a947-123bd85a4968","Type":"ContainerStarted","Data":"4f15f34f28e86ff388667b9f6ae2d0c86f1c6931168dfb4f72443c054797aef8"}
Oct 30 04:57:12.278560 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:12.278493    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49" podStartSLOduration=2.201349418 podStartE2EDuration="3.278476753s" totalImagesPullingTime="1.077127335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:57:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:57:12.277530528 +0000 UTC m=+761.501350428" watchObservedRunningTime="2026-10-30 04:57:12.278476753 +0000 UTC m=+761.502296655"
Oct 30 04:57:12.829694 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:12.829664    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:57:13.106529 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:13.106429    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49"
Oct 30 04:57:13.510929 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:13.510831    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 04:57:14.703863 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:14.703831    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:57:15.911454 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:15.911423    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:57:16.011511 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:16.011472    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:57:16.112367 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:16.112329    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:57:16.584117 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:16.584077    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:57:16.687999 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:16.687969    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:57:16.787915 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:16.787877    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:57:16.891208 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:16.891129    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:57:18.165987 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:18.165953    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:57:18.713478 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:18.713439    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:57:19.770917 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:19.770873    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:57:20.107249 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:20.107217    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49"
Oct 30 04:57:20.119868 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:20.119843    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49"
Oct 30 04:57:20.291717 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:20.291681    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49"
Oct 30 04:57:20.390361 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:20.390285    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:57:21.994767 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:21.994717    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 04:57:22.093850 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:22.093813    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx"]
Oct 30 04:57:22.098084 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:22.098058    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx"
Oct 30 04:57:22.106472 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:22.106444    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:57:22.114845 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:22.114818    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx"]
Oct 30 04:57:22.124503 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:22.124482    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b21507\"/\"simulator-w0-b21507-kserve-self-signed-certs\""
Oct 30 04:57:22.124610 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:22.124508    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-b21507\"/\"openshift-service-ca.crt\""
Oct 30 04:57:22.124610 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:22.124548    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-b21507\"/\"kube-root-ca.crt\""
Oct 30 04:57:22.124698 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:22.124664    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b21507\"/\"default-dockercfg-9drgk\""
Oct 30 04:57:22.161260 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:22.161229    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d523008f-c740-4a41-89cb-e82e1ffa0094-tls-certs\") pod \"simulator-w0-b21507-kserve-5f4f779998-7msgx\" (UID: \"d523008f-c740-4a41-89cb-e82e1ffa0094\") " pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx"
Oct 30 04:57:22.161384 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:22.161271    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d523008f-c740-4a41-89cb-e82e1ffa0094-model-cache\") pod \"simulator-w0-b21507-kserve-5f4f779998-7msgx\" (UID: \"d523008f-c740-4a41-89cb-e82e1ffa0094\") " pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx"
Oct 30 04:57:22.161384 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:22.161344    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d523008f-c740-4a41-89cb-e82e1ffa0094-tmp-dir\") pod \"simulator-w0-b21507-kserve-5f4f779998-7msgx\" (UID: \"d523008f-c740-4a41-89cb-e82e1ffa0094\") " pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx"
Oct 30 04:57:22.161507 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:22.161385    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hrhg\" (UniqueName: \"kubernetes.io/projected/d523008f-c740-4a41-89cb-e82e1ffa0094-kube-api-access-4hrhg\") pod \"simulator-w0-b21507-kserve-5f4f779998-7msgx\" (UID: \"d523008f-c740-4a41-89cb-e82e1ffa0094\") " pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx"
Oct 30 04:57:22.161507 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:22.161497    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d523008f-c740-4a41-89cb-e82e1ffa0094-dshm\") pod \"simulator-w0-b21507-kserve-5f4f779998-7msgx\" (UID: \"d523008f-c740-4a41-89cb-e82e1ffa0094\") " pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx"
Oct 30 04:57:22.161620 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:22.161523    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d523008f-c740-4a41-89cb-e82e1ffa0094-home\") pod \"simulator-w0-b21507-kserve-5f4f779998-7msgx\" (UID: \"d523008f-c740-4a41-89cb-e82e1ffa0094\") " pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx"
Oct 30 04:57:22.203697 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:22.203658    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:57:22.262300 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:22.262208    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d523008f-c740-4a41-89cb-e82e1ffa0094-dshm\") pod \"simulator-w0-b21507-kserve-5f4f779998-7msgx\" (UID: \"d523008f-c740-4a41-89cb-e82e1ffa0094\") " pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx"
Oct 30 04:57:22.262300 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:22.262263    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d523008f-c740-4a41-89cb-e82e1ffa0094-home\") pod \"simulator-w0-b21507-kserve-5f4f779998-7msgx\" (UID: \"d523008f-c740-4a41-89cb-e82e1ffa0094\") " pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx"
Oct 30 04:57:22.262521 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:22.262354    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d523008f-c740-4a41-89cb-e82e1ffa0094-tls-certs\") pod \"simulator-w0-b21507-kserve-5f4f779998-7msgx\" (UID: \"d523008f-c740-4a41-89cb-e82e1ffa0094\") " pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx"
Oct 30 04:57:22.262521 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:22.262380    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d523008f-c740-4a41-89cb-e82e1ffa0094-model-cache\") pod \"simulator-w0-b21507-kserve-5f4f779998-7msgx\" (UID: \"d523008f-c740-4a41-89cb-e82e1ffa0094\") " pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx"
Oct 30 04:57:22.262521 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:22.262404    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d523008f-c740-4a41-89cb-e82e1ffa0094-tmp-dir\") pod \"simulator-w0-b21507-kserve-5f4f779998-7msgx\" (UID: \"d523008f-c740-4a41-89cb-e82e1ffa0094\") " pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx"
Oct 30 04:57:22.262521 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:22.262440    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hrhg\" (UniqueName: \"kubernetes.io/projected/d523008f-c740-4a41-89cb-e82e1ffa0094-kube-api-access-4hrhg\") pod \"simulator-w0-b21507-kserve-5f4f779998-7msgx\" (UID: \"d523008f-c740-4a41-89cb-e82e1ffa0094\") " pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx"
Oct 30 04:57:22.262800 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:22.262764    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d523008f-c740-4a41-89cb-e82e1ffa0094-home\") pod \"simulator-w0-b21507-kserve-5f4f779998-7msgx\" (UID: \"d523008f-c740-4a41-89cb-e82e1ffa0094\") " pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx"
Oct 30 04:57:22.262972 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:22.262953    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d523008f-c740-4a41-89cb-e82e1ffa0094-tmp-dir\") pod \"simulator-w0-b21507-kserve-5f4f779998-7msgx\" (UID: \"d523008f-c740-4a41-89cb-e82e1ffa0094\") " pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx"
Oct 30 04:57:22.263051 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:22.263036    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d523008f-c740-4a41-89cb-e82e1ffa0094-model-cache\") pod \"simulator-w0-b21507-kserve-5f4f779998-7msgx\" (UID: \"d523008f-c740-4a41-89cb-e82e1ffa0094\") " pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx"
Oct 30 04:57:22.264554 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:22.264526    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d523008f-c740-4a41-89cb-e82e1ffa0094-dshm\") pod \"simulator-w0-b21507-kserve-5f4f779998-7msgx\" (UID: \"d523008f-c740-4a41-89cb-e82e1ffa0094\") " pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx"
Oct 30 04:57:22.265225 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:22.265204    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d523008f-c740-4a41-89cb-e82e1ffa0094-tls-certs\") pod \"simulator-w0-b21507-kserve-5f4f779998-7msgx\" (UID: \"d523008f-c740-4a41-89cb-e82e1ffa0094\") " pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx"
Oct 30 04:57:22.288964 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:22.288933    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hrhg\" (UniqueName: \"kubernetes.io/projected/d523008f-c740-4a41-89cb-e82e1ffa0094-kube-api-access-4hrhg\") pod \"simulator-w0-b21507-kserve-5f4f779998-7msgx\" (UID: \"d523008f-c740-4a41-89cb-e82e1ffa0094\") " pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx"
Oct 30 04:57:22.410825 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:22.410789    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx"
Oct 30 04:57:22.718699 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:22.718666    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx"]
Oct 30 04:57:22.722174 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:57:22.722143    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd523008f_c740_4a41_89cb_e82e1ffa0094.slice/crio-990bb437d80bbd6999f9c10cebf9c13c1db4eeb7a2a4d714fe80601526764c7f WatchSource:0}: Error finding container 990bb437d80bbd6999f9c10cebf9c13c1db4eeb7a2a4d714fe80601526764c7f: Status 404 returned error can't find the container with id 990bb437d80bbd6999f9c10cebf9c13c1db4eeb7a2a4d714fe80601526764c7f
Oct 30 04:57:23.300789 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:23.300748    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx" event={"ID":"d523008f-c740-4a41-89cb-e82e1ffa0094","Type":"ContainerStarted","Data":"09527f87c929c6f43806add434b18df957d709cc967f2aba2be1e376fc61e00f"}
Oct 30 04:57:23.300789 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:23.300789    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx" event={"ID":"d523008f-c740-4a41-89cb-e82e1ffa0094","Type":"ContainerStarted","Data":"990bb437d80bbd6999f9c10cebf9c13c1db4eeb7a2a4d714fe80601526764c7f"}
Oct 30 04:57:23.339484 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:23.339435    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx" podStartSLOduration=1.339420804 podStartE2EDuration="1.339420804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:57:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:57:23.337051728 +0000 UTC m=+772.560871630" watchObservedRunningTime="2026-10-30 04:57:23.339420804 +0000 UTC m=+772.563240737"
Oct 30 04:57:23.411424 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:23.411389    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx"
Oct 30 04:57:23.825107 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:23.825077    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:57:25.484561 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:25.484517    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:57:25.942572 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:25.942536    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:57:26.817731 ip-10-0-135-98 kubenswrapper[5898]: E1030 04:57:26.817693    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 04:57:27.363836 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:27.363806    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:57:28.062065 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:28.062033    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9"]
Oct 30 04:57:28.070274 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:28.070249    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9"
Oct 30 04:57:28.083530 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:28.083503    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9"]
Oct 30 04:57:28.093352 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:28.093328    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-d55037\"/\"default-dockercfg-w4dgw\""
Oct 30 04:57:28.093708 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:28.093691    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-d55037\"/\"simulator-w4-d55037-kserve-self-signed-certs\""
Oct 30 04:57:28.096733 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:28.096717    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-d55037\"/\"kube-root-ca.crt\""
Oct 30 04:57:28.096803 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:28.096752    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-d55037\"/\"openshift-service-ca.crt\""
Oct 30 04:57:28.113093 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:28.113067    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1647f909-5580-4219-a2e1-30b4cd994379-dshm\") pod \"simulator-w4-d55037-kserve-6b959b69bc-cjlv9\" (UID: \"1647f909-5580-4219-a2e1-30b4cd994379\") " pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9"
Oct 30 04:57:28.113229 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:28.113116    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vrqp\" (UniqueName: \"kubernetes.io/projected/1647f909-5580-4219-a2e1-30b4cd994379-kube-api-access-2vrqp\") pod \"simulator-w4-d55037-kserve-6b959b69bc-cjlv9\" (UID: \"1647f909-5580-4219-a2e1-30b4cd994379\") " pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9"
Oct 30 04:57:28.113306 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:28.113262    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1647f909-5580-4219-a2e1-30b4cd994379-model-cache\") pod \"simulator-w4-d55037-kserve-6b959b69bc-cjlv9\" (UID: \"1647f909-5580-4219-a2e1-30b4cd994379\") " pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9"
Oct 30 04:57:28.113306 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:28.113300    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1647f909-5580-4219-a2e1-30b4cd994379-tls-certs\") pod \"simulator-w4-d55037-kserve-6b959b69bc-cjlv9\" (UID: \"1647f909-5580-4219-a2e1-30b4cd994379\") " pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9"
Oct 30 04:57:28.113428 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:28.113410    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1647f909-5580-4219-a2e1-30b4cd994379-home\") pod \"simulator-w4-d55037-kserve-6b959b69bc-cjlv9\" (UID: \"1647f909-5580-4219-a2e1-30b4cd994379\") " pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9"
Oct 30 04:57:28.113467 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:28.113456    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1647f909-5580-4219-a2e1-30b4cd994379-tmp-dir\") pod \"simulator-w4-d55037-kserve-6b959b69bc-cjlv9\" (UID: \"1647f909-5580-4219-a2e1-30b4cd994379\") " pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9"
Oct 30 04:57:28.119110 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:28.119087    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:57:28.213877 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:28.213842    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1647f909-5580-4219-a2e1-30b4cd994379-model-cache\") pod \"simulator-w4-d55037-kserve-6b959b69bc-cjlv9\" (UID: \"1647f909-5580-4219-a2e1-30b4cd994379\") " pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9"
Oct 30 04:57:28.213877 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:28.213880    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1647f909-5580-4219-a2e1-30b4cd994379-tls-certs\") pod \"simulator-w4-d55037-kserve-6b959b69bc-cjlv9\" (UID: \"1647f909-5580-4219-a2e1-30b4cd994379\") " pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9"
Oct 30 04:57:28.214118 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:28.214016    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1647f909-5580-4219-a2e1-30b4cd994379-home\") pod \"simulator-w4-d55037-kserve-6b959b69bc-cjlv9\" (UID: \"1647f909-5580-4219-a2e1-30b4cd994379\") " pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9"
Oct 30 04:57:28.214118 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:28.214077    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1647f909-5580-4219-a2e1-30b4cd994379-tmp-dir\") pod \"simulator-w4-d55037-kserve-6b959b69bc-cjlv9\" (UID: \"1647f909-5580-4219-a2e1-30b4cd994379\") " pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9"
Oct 30 04:57:28.214223 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:28.214128    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1647f909-5580-4219-a2e1-30b4cd994379-dshm\") pod \"simulator-w4-d55037-kserve-6b959b69bc-cjlv9\" (UID: \"1647f909-5580-4219-a2e1-30b4cd994379\") " pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9"
Oct 30 04:57:28.214223 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:28.214164    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vrqp\" (UniqueName: \"kubernetes.io/projected/1647f909-5580-4219-a2e1-30b4cd994379-kube-api-access-2vrqp\") pod \"simulator-w4-d55037-kserve-6b959b69bc-cjlv9\" (UID: \"1647f909-5580-4219-a2e1-30b4cd994379\") " pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9"
Oct 30 04:57:28.214358 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:28.214338    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1647f909-5580-4219-a2e1-30b4cd994379-model-cache\") pod \"simulator-w4-d55037-kserve-6b959b69bc-cjlv9\" (UID: \"1647f909-5580-4219-a2e1-30b4cd994379\") " pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9"
Oct 30 04:57:28.214412 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:28.214365    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1647f909-5580-4219-a2e1-30b4cd994379-home\") pod \"simulator-w4-d55037-kserve-6b959b69bc-cjlv9\" (UID: \"1647f909-5580-4219-a2e1-30b4cd994379\") " pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9"
Oct 30 04:57:28.214458 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:28.214436    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1647f909-5580-4219-a2e1-30b4cd994379-tmp-dir\") pod \"simulator-w4-d55037-kserve-6b959b69bc-cjlv9\" (UID: \"1647f909-5580-4219-a2e1-30b4cd994379\") " pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9"
Oct 30 04:57:28.216416 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:28.216389    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1647f909-5580-4219-a2e1-30b4cd994379-dshm\") pod \"simulator-w4-d55037-kserve-6b959b69bc-cjlv9\" (UID: \"1647f909-5580-4219-a2e1-30b4cd994379\") " pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9"
Oct 30 04:57:28.216535 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:28.216517    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1647f909-5580-4219-a2e1-30b4cd994379-tls-certs\") pod \"simulator-w4-d55037-kserve-6b959b69bc-cjlv9\" (UID: \"1647f909-5580-4219-a2e1-30b4cd994379\") " pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9"
Oct 30 04:57:28.232507 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:28.232484    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vrqp\" (UniqueName: \"kubernetes.io/projected/1647f909-5580-4219-a2e1-30b4cd994379-kube-api-access-2vrqp\") pod \"simulator-w4-d55037-kserve-6b959b69bc-cjlv9\" (UID: \"1647f909-5580-4219-a2e1-30b4cd994379\") " pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9"
Oct 30 04:57:28.385402 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:28.385370    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9"
Oct 30 04:57:28.676782 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:28.676696    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9"]
Oct 30 04:57:28.679641 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:57:28.679599    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1647f909_5580_4219_a2e1_30b4cd994379.slice/crio-ecdd6f593bb7feda4576614f45ebe9d16a45c3546348e422117a57bab77ac287 WatchSource:0}: Error finding container ecdd6f593bb7feda4576614f45ebe9d16a45c3546348e422117a57bab77ac287: Status 404 returned error can't find the container with id ecdd6f593bb7feda4576614f45ebe9d16a45c3546348e422117a57bab77ac287
Oct 30 04:57:29.059137 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:29.059058    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:57:29.332485 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:29.332454    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9" event={"ID":"1647f909-5580-4219-a2e1-30b4cd994379","Type":"ContainerStarted","Data":"70240ed3054547cf492254df2362e4f9c35094c6497d02127da86c58a817074d"}
Oct 30 04:57:29.332978 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:29.332495    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9" event={"ID":"1647f909-5580-4219-a2e1-30b4cd994379","Type":"ContainerStarted","Data":"ecdd6f593bb7feda4576614f45ebe9d16a45c3546348e422117a57bab77ac287"}
Oct 30 04:57:29.371586 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:29.371537    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9" podStartSLOduration=1.37152468 podStartE2EDuration="1.37152468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:57:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:57:29.370886819 +0000 UTC m=+778.594706738" watchObservedRunningTime="2026-10-30 04:57:29.37152468 +0000 UTC m=+778.595344581"
Oct 30 04:57:29.386409 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:29.386380    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9"
Oct 30 04:57:30.739614 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:30.739570    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:57:32.336102 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:32.336069    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:57:32.411170 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:32.411138    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx"
Oct 30 04:57:32.424227 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:32.424200    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx"
Oct 30 04:57:32.577391 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:32.577355    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 04:57:33.370316 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:33.370287    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx"
Oct 30 04:57:34.029156 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:34.029124    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:57:35.746210 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:35.746175    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:57:35.893472 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:35.893436    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58"]
Oct 30 04:57:35.901275 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:35.901243    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58"
Oct 30 04:57:35.919986 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:35.919956    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58"]
Oct 30 04:57:35.928083 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:35.928043    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-83e83e\"/\"openshift-service-ca.crt\""
Oct 30 04:57:35.928083 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:35.928082    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-83e83e\"/\"default-dockercfg-m9psh\""
Oct 30 04:57:35.928258 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:35.928088    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-83e83e\"/\"kube-root-ca.crt\""
Oct 30 04:57:35.928258 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:35.928183    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-83e83e\"/\"simulator-w6-83e83e-kserve-self-signed-certs\""
Oct 30 04:57:35.944388 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:35.944358    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:57:35.991871 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:35.991844    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c27f3b21-1447-4af9-8d99-c04fa9b23cda-home\") pod \"simulator-w6-83e83e-kserve-f76fb6869-qjk58\" (UID: \"c27f3b21-1447-4af9-8d99-c04fa9b23cda\") " pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58"
Oct 30 04:57:35.992044 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:35.991894    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c27f3b21-1447-4af9-8d99-c04fa9b23cda-tmp-dir\") pod \"simulator-w6-83e83e-kserve-f76fb6869-qjk58\" (UID: \"c27f3b21-1447-4af9-8d99-c04fa9b23cda\") " pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58"
Oct 30 04:57:35.992044 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:35.991998    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c27f3b21-1447-4af9-8d99-c04fa9b23cda-model-cache\") pod \"simulator-w6-83e83e-kserve-f76fb6869-qjk58\" (UID: \"c27f3b21-1447-4af9-8d99-c04fa9b23cda\") " pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58"
Oct 30 04:57:35.992160 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:35.992039    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c27f3b21-1447-4af9-8d99-c04fa9b23cda-tls-certs\") pod \"simulator-w6-83e83e-kserve-f76fb6869-qjk58\" (UID: \"c27f3b21-1447-4af9-8d99-c04fa9b23cda\") " pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58"
Oct 30 04:57:35.992160 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:35.992072    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92kgx\" (UniqueName: \"kubernetes.io/projected/c27f3b21-1447-4af9-8d99-c04fa9b23cda-kube-api-access-92kgx\") pod \"simulator-w6-83e83e-kserve-f76fb6869-qjk58\" (UID: \"c27f3b21-1447-4af9-8d99-c04fa9b23cda\") " pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58"
Oct 30 04:57:35.992267 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:35.992190    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c27f3b21-1447-4af9-8d99-c04fa9b23cda-dshm\") pod \"simulator-w6-83e83e-kserve-f76fb6869-qjk58\" (UID: \"c27f3b21-1447-4af9-8d99-c04fa9b23cda\") " pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58"
Oct 30 04:57:36.093462 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:36.093423    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c27f3b21-1447-4af9-8d99-c04fa9b23cda-dshm\") pod \"simulator-w6-83e83e-kserve-f76fb6869-qjk58\" (UID: \"c27f3b21-1447-4af9-8d99-c04fa9b23cda\") " pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58"
Oct 30 04:57:36.093656 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:36.093555    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c27f3b21-1447-4af9-8d99-c04fa9b23cda-home\") pod \"simulator-w6-83e83e-kserve-f76fb6869-qjk58\" (UID: \"c27f3b21-1447-4af9-8d99-c04fa9b23cda\") " pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58"
Oct 30 04:57:36.093656 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:36.093622    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c27f3b21-1447-4af9-8d99-c04fa9b23cda-tmp-dir\") pod \"simulator-w6-83e83e-kserve-f76fb6869-qjk58\" (UID: \"c27f3b21-1447-4af9-8d99-c04fa9b23cda\") " pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58"
Oct 30 04:57:36.093773 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:36.093663    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c27f3b21-1447-4af9-8d99-c04fa9b23cda-model-cache\") pod \"simulator-w6-83e83e-kserve-f76fb6869-qjk58\" (UID: \"c27f3b21-1447-4af9-8d99-c04fa9b23cda\") " pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58"
Oct 30 04:57:36.093773 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:36.093698    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c27f3b21-1447-4af9-8d99-c04fa9b23cda-tls-certs\") pod \"simulator-w6-83e83e-kserve-f76fb6869-qjk58\" (UID: \"c27f3b21-1447-4af9-8d99-c04fa9b23cda\") " pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58"
Oct 30 04:57:36.093773 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:36.093721    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92kgx\" (UniqueName: \"kubernetes.io/projected/c27f3b21-1447-4af9-8d99-c04fa9b23cda-kube-api-access-92kgx\") pod \"simulator-w6-83e83e-kserve-f76fb6869-qjk58\" (UID: \"c27f3b21-1447-4af9-8d99-c04fa9b23cda\") " pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58"
Oct 30 04:57:36.094034 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:36.094009    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c27f3b21-1447-4af9-8d99-c04fa9b23cda-home\") pod \"simulator-w6-83e83e-kserve-f76fb6869-qjk58\" (UID: \"c27f3b21-1447-4af9-8d99-c04fa9b23cda\") " pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58"
Oct 30 04:57:36.094118 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:36.094046    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c27f3b21-1447-4af9-8d99-c04fa9b23cda-tmp-dir\") pod \"simulator-w6-83e83e-kserve-f76fb6869-qjk58\" (UID: \"c27f3b21-1447-4af9-8d99-c04fa9b23cda\") " pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58"
Oct 30 04:57:36.094118 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:36.094105    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c27f3b21-1447-4af9-8d99-c04fa9b23cda-model-cache\") pod \"simulator-w6-83e83e-kserve-f76fb6869-qjk58\" (UID: \"c27f3b21-1447-4af9-8d99-c04fa9b23cda\") " pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58"
Oct 30 04:57:36.096283 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:36.096262    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c27f3b21-1447-4af9-8d99-c04fa9b23cda-dshm\") pod \"simulator-w6-83e83e-kserve-f76fb6869-qjk58\" (UID: \"c27f3b21-1447-4af9-8d99-c04fa9b23cda\") " pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58"
Oct 30 04:57:36.096419 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:36.096391    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c27f3b21-1447-4af9-8d99-c04fa9b23cda-tls-certs\") pod \"simulator-w6-83e83e-kserve-f76fb6869-qjk58\" (UID: \"c27f3b21-1447-4af9-8d99-c04fa9b23cda\") " pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58"
Oct 30 04:57:36.114006 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:36.113978    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92kgx\" (UniqueName: \"kubernetes.io/projected/c27f3b21-1447-4af9-8d99-c04fa9b23cda-kube-api-access-92kgx\") pod \"simulator-w6-83e83e-kserve-f76fb6869-qjk58\" (UID: \"c27f3b21-1447-4af9-8d99-c04fa9b23cda\") " pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58"
Oct 30 04:57:36.215170 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:36.215135    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58"
Oct 30 04:57:36.513529 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:36.513504    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58"]
Oct 30 04:57:36.515052 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:57:36.515026    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc27f3b21_1447_4af9_8d99_c04fa9b23cda.slice/crio-1a6cfdcd44c131c15223e5be73a5a7107d08f26d2e869ec8b6df374f7c9cfd32 WatchSource:0}: Error finding container 1a6cfdcd44c131c15223e5be73a5a7107d08f26d2e869ec8b6df374f7c9cfd32: Status 404 returned error can't find the container with id 1a6cfdcd44c131c15223e5be73a5a7107d08f26d2e869ec8b6df374f7c9cfd32
Oct 30 04:57:37.375198 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:37.375162    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58" event={"ID":"c27f3b21-1447-4af9-8d99-c04fa9b23cda","Type":"ContainerStarted","Data":"ae0c691c6bc69ffdb23781f20e0eedf669f942c27bd990c10a039f10472ce2c7"}
Oct 30 04:57:37.375688 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:37.375207    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58" event={"ID":"c27f3b21-1447-4af9-8d99-c04fa9b23cda","Type":"ContainerStarted","Data":"1a6cfdcd44c131c15223e5be73a5a7107d08f26d2e869ec8b6df374f7c9cfd32"}
Oct 30 04:57:37.412939 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:37.412882    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58" podStartSLOduration=2.412869135 podStartE2EDuration="2.412869135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:57:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:57:37.412141464 +0000 UTC m=+786.635961365" watchObservedRunningTime="2026-10-30 04:57:37.412869135 +0000 UTC m=+786.636689035"
Oct 30 04:57:37.439981 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:37.439952    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:57:37.992236 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:37.992204    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:57:38.215369 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:38.215336    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58"
Oct 30 04:57:38.386376 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:38.386345    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9"
Oct 30 04:57:38.398860 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:38.398836    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9"
Oct 30 04:57:39.111786 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:39.111732    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:57:39.396889 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:39.396811    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9"
Oct 30 04:57:40.881026 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:40.880985    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:57:42.133661 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:42.133620    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb"]
Oct 30 04:57:42.140023 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:42.139994    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb"
Oct 30 04:57:42.154864 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:42.154840    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb"]
Oct 30 04:57:42.164448 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:42.164423    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-397179\"/\"premium-w2-397179-kserve-self-signed-certs\""
Oct 30 04:57:42.170014 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:42.169989    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:57:42.257331 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:42.257302    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f16bd6a-a7f0-49bb-844f-57b777bde2ed-dshm\") pod \"premium-w2-397179-kserve-797dbcc7bd-d9jwb\" (UID: \"6f16bd6a-a7f0-49bb-844f-57b777bde2ed\") " pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb"
Oct 30 04:57:42.257510 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:42.257340    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j75z\" (UniqueName: \"kubernetes.io/projected/6f16bd6a-a7f0-49bb-844f-57b777bde2ed-kube-api-access-2j75z\") pod \"premium-w2-397179-kserve-797dbcc7bd-d9jwb\" (UID: \"6f16bd6a-a7f0-49bb-844f-57b777bde2ed\") " pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb"
Oct 30 04:57:42.257510 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:42.257377    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f16bd6a-a7f0-49bb-844f-57b777bde2ed-tls-certs\") pod \"premium-w2-397179-kserve-797dbcc7bd-d9jwb\" (UID: \"6f16bd6a-a7f0-49bb-844f-57b777bde2ed\") " pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb"
Oct 30 04:57:42.257597 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:42.257535    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f16bd6a-a7f0-49bb-844f-57b777bde2ed-model-cache\") pod \"premium-w2-397179-kserve-797dbcc7bd-d9jwb\" (UID: \"6f16bd6a-a7f0-49bb-844f-57b777bde2ed\") " pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb"
Oct 30 04:57:42.257597 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:42.257576    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f16bd6a-a7f0-49bb-844f-57b777bde2ed-home\") pod \"premium-w2-397179-kserve-797dbcc7bd-d9jwb\" (UID: \"6f16bd6a-a7f0-49bb-844f-57b777bde2ed\") " pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb"
Oct 30 04:57:42.257700 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:42.257602    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f16bd6a-a7f0-49bb-844f-57b777bde2ed-tmp-dir\") pod \"premium-w2-397179-kserve-797dbcc7bd-d9jwb\" (UID: \"6f16bd6a-a7f0-49bb-844f-57b777bde2ed\") " pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb"
Oct 30 04:57:42.359058 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:42.359019    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j75z\" (UniqueName: \"kubernetes.io/projected/6f16bd6a-a7f0-49bb-844f-57b777bde2ed-kube-api-access-2j75z\") pod \"premium-w2-397179-kserve-797dbcc7bd-d9jwb\" (UID: \"6f16bd6a-a7f0-49bb-844f-57b777bde2ed\") " pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb"
Oct 30 04:57:42.359257 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:42.359074    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f16bd6a-a7f0-49bb-844f-57b777bde2ed-tls-certs\") pod \"premium-w2-397179-kserve-797dbcc7bd-d9jwb\" (UID: \"6f16bd6a-a7f0-49bb-844f-57b777bde2ed\") " pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb"
Oct 30 04:57:42.359257 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:42.359103    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f16bd6a-a7f0-49bb-844f-57b777bde2ed-model-cache\") pod \"premium-w2-397179-kserve-797dbcc7bd-d9jwb\" (UID: \"6f16bd6a-a7f0-49bb-844f-57b777bde2ed\") " pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb"
Oct 30 04:57:42.359257 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:42.359121    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f16bd6a-a7f0-49bb-844f-57b777bde2ed-home\") pod \"premium-w2-397179-kserve-797dbcc7bd-d9jwb\" (UID: \"6f16bd6a-a7f0-49bb-844f-57b777bde2ed\") " pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb"
Oct 30 04:57:42.359257 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:42.359136    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f16bd6a-a7f0-49bb-844f-57b777bde2ed-tmp-dir\") pod \"premium-w2-397179-kserve-797dbcc7bd-d9jwb\" (UID: \"6f16bd6a-a7f0-49bb-844f-57b777bde2ed\") " pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb"
Oct 30 04:57:42.359471 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:42.359273    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f16bd6a-a7f0-49bb-844f-57b777bde2ed-dshm\") pod \"premium-w2-397179-kserve-797dbcc7bd-d9jwb\" (UID: \"6f16bd6a-a7f0-49bb-844f-57b777bde2ed\") " pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb"
Oct 30 04:57:42.359568 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:42.359548    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f16bd6a-a7f0-49bb-844f-57b777bde2ed-home\") pod \"premium-w2-397179-kserve-797dbcc7bd-d9jwb\" (UID: \"6f16bd6a-a7f0-49bb-844f-57b777bde2ed\") " pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb"
Oct 30 04:57:42.359625 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:42.359594    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f16bd6a-a7f0-49bb-844f-57b777bde2ed-model-cache\") pod \"premium-w2-397179-kserve-797dbcc7bd-d9jwb\" (UID: \"6f16bd6a-a7f0-49bb-844f-57b777bde2ed\") " pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb"
Oct 30 04:57:42.359678 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:42.359620    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f16bd6a-a7f0-49bb-844f-57b777bde2ed-tmp-dir\") pod \"premium-w2-397179-kserve-797dbcc7bd-d9jwb\" (UID: \"6f16bd6a-a7f0-49bb-844f-57b777bde2ed\") " pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb"
Oct 30 04:57:42.361562 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:42.361534    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f16bd6a-a7f0-49bb-844f-57b777bde2ed-dshm\") pod \"premium-w2-397179-kserve-797dbcc7bd-d9jwb\" (UID: \"6f16bd6a-a7f0-49bb-844f-57b777bde2ed\") " pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb"
Oct 30 04:57:42.361775 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:42.361756    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f16bd6a-a7f0-49bb-844f-57b777bde2ed-tls-certs\") pod \"premium-w2-397179-kserve-797dbcc7bd-d9jwb\" (UID: \"6f16bd6a-a7f0-49bb-844f-57b777bde2ed\") " pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb"
Oct 30 04:57:42.376941 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:42.376892    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j75z\" (UniqueName: \"kubernetes.io/projected/6f16bd6a-a7f0-49bb-844f-57b777bde2ed-kube-api-access-2j75z\") pod \"premium-w2-397179-kserve-797dbcc7bd-d9jwb\" (UID: \"6f16bd6a-a7f0-49bb-844f-57b777bde2ed\") " pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb"
Oct 30 04:57:42.452366 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:42.452279    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb"
Oct 30 04:57:42.586656 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:42.586617    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:57:42.749233 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:42.749207    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb"]
Oct 30 04:57:42.751051 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:57:42.751026    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6f16bd6a_a7f0_49bb_844f_57b777bde2ed.slice/crio-cdf000ba236a68d99a0632777c9d516f67bc0982c56d66a6cb8467c0e0134188 WatchSource:0}: Error finding container cdf000ba236a68d99a0632777c9d516f67bc0982c56d66a6cb8467c0e0134188: Status 404 returned error can't find the container with id cdf000ba236a68d99a0632777c9d516f67bc0982c56d66a6cb8467c0e0134188
Oct 30 04:57:43.407859 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:43.407823    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb" event={"ID":"6f16bd6a-a7f0-49bb-844f-57b777bde2ed","Type":"ContainerStarted","Data":"9e43b28fab1725efc5f8953ba3c552342e7224362b718c643f783f26beb042b2"}
Oct 30 04:57:43.407859 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:43.407872    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb" event={"ID":"6f16bd6a-a7f0-49bb-844f-57b777bde2ed","Type":"ContainerStarted","Data":"cdf000ba236a68d99a0632777c9d516f67bc0982c56d66a6cb8467c0e0134188"}
Oct 30 04:57:43.444571 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:43.444528    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb" podStartSLOduration=1.4445136920000001 podStartE2EDuration="1.444513692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:57:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:57:43.444278252 +0000 UTC m=+792.668098165" watchObservedRunningTime="2026-10-30 04:57:43.444513692 +0000 UTC m=+792.668333594"
Oct 30 04:57:43.453376 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:43.453349    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb"
Oct 30 04:57:44.321082 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:44.321046    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:57:45.909058 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:45.909019    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:57:46.216110 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:46.216032    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58"
Oct 30 04:57:46.228497 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:46.228471    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58"
Oct 30 04:57:46.294579 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:46.294553    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 04:57:46.434475 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:46.434446    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58"
Oct 30 04:57:47.633296 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:47.631782    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:57:49.391642 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:49.391601    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:57:50.963129 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:50.963095    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:57:51.986002 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:51.985969    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh"]
Oct 30 04:57:51.990569 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:51.990552    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh"
Oct 30 04:57:52.016484 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:52.016458    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b21507\"/\"premium-w0-b21507-kserve-self-signed-certs\""
Oct 30 04:57:52.018852 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:52.018826    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh"]
Oct 30 04:57:52.040689 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:52.040664    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:57:52.156698 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:52.156660    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38428a53-04c7-400a-9c78-52407e982478-model-cache\") pod \"premium-w0-b21507-kserve-75db566f55-sxwvh\" (UID: \"38428a53-04c7-400a-9c78-52407e982478\") " pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh"
Oct 30 04:57:52.156894 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:52.156714    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38428a53-04c7-400a-9c78-52407e982478-dshm\") pod \"premium-w0-b21507-kserve-75db566f55-sxwvh\" (UID: \"38428a53-04c7-400a-9c78-52407e982478\") " pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh"
Oct 30 04:57:52.156894 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:52.156824    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38428a53-04c7-400a-9c78-52407e982478-tmp-dir\") pod \"premium-w0-b21507-kserve-75db566f55-sxwvh\" (UID: \"38428a53-04c7-400a-9c78-52407e982478\") " pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh"
Oct 30 04:57:52.156894 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:52.156857    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrmnz\" (UniqueName: \"kubernetes.io/projected/38428a53-04c7-400a-9c78-52407e982478-kube-api-access-zrmnz\") pod \"premium-w0-b21507-kserve-75db566f55-sxwvh\" (UID: \"38428a53-04c7-400a-9c78-52407e982478\") " pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh"
Oct 30 04:57:52.157097 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:52.156932    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38428a53-04c7-400a-9c78-52407e982478-tls-certs\") pod \"premium-w0-b21507-kserve-75db566f55-sxwvh\" (UID: \"38428a53-04c7-400a-9c78-52407e982478\") " pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh"
Oct 30 04:57:52.157097 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:52.156996    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38428a53-04c7-400a-9c78-52407e982478-home\") pod \"premium-w0-b21507-kserve-75db566f55-sxwvh\" (UID: \"38428a53-04c7-400a-9c78-52407e982478\") " pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh"
Oct 30 04:57:52.258574 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:52.258473    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38428a53-04c7-400a-9c78-52407e982478-dshm\") pod \"premium-w0-b21507-kserve-75db566f55-sxwvh\" (UID: \"38428a53-04c7-400a-9c78-52407e982478\") " pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh"
Oct 30 04:57:52.258574 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:52.258529    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38428a53-04c7-400a-9c78-52407e982478-tmp-dir\") pod \"premium-w0-b21507-kserve-75db566f55-sxwvh\" (UID: \"38428a53-04c7-400a-9c78-52407e982478\") " pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh"
Oct 30 04:57:52.258574 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:52.258552    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrmnz\" (UniqueName: \"kubernetes.io/projected/38428a53-04c7-400a-9c78-52407e982478-kube-api-access-zrmnz\") pod \"premium-w0-b21507-kserve-75db566f55-sxwvh\" (UID: \"38428a53-04c7-400a-9c78-52407e982478\") " pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh"
Oct 30 04:57:52.258872 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:52.258578    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38428a53-04c7-400a-9c78-52407e982478-tls-certs\") pod \"premium-w0-b21507-kserve-75db566f55-sxwvh\" (UID: \"38428a53-04c7-400a-9c78-52407e982478\") " pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh"
Oct 30 04:57:52.258872 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:52.258609    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38428a53-04c7-400a-9c78-52407e982478-home\") pod \"premium-w0-b21507-kserve-75db566f55-sxwvh\" (UID: \"38428a53-04c7-400a-9c78-52407e982478\") " pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh"
Oct 30 04:57:52.258872 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:52.258707    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38428a53-04c7-400a-9c78-52407e982478-model-cache\") pod \"premium-w0-b21507-kserve-75db566f55-sxwvh\" (UID: \"38428a53-04c7-400a-9c78-52407e982478\") " pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh"
Oct 30 04:57:52.259014 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:52.258998    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38428a53-04c7-400a-9c78-52407e982478-tmp-dir\") pod \"premium-w0-b21507-kserve-75db566f55-sxwvh\" (UID: \"38428a53-04c7-400a-9c78-52407e982478\") " pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh"
Oct 30 04:57:52.259090 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:52.259065    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38428a53-04c7-400a-9c78-52407e982478-home\") pod \"premium-w0-b21507-kserve-75db566f55-sxwvh\" (UID: \"38428a53-04c7-400a-9c78-52407e982478\") " pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh"
Oct 30 04:57:52.259175 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:52.259144    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38428a53-04c7-400a-9c78-52407e982478-model-cache\") pod \"premium-w0-b21507-kserve-75db566f55-sxwvh\" (UID: \"38428a53-04c7-400a-9c78-52407e982478\") " pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh"
Oct 30 04:57:52.260862 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:52.260842    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38428a53-04c7-400a-9c78-52407e982478-dshm\") pod \"premium-w0-b21507-kserve-75db566f55-sxwvh\" (UID: \"38428a53-04c7-400a-9c78-52407e982478\") " pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh"
Oct 30 04:57:52.261299 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:52.261284    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38428a53-04c7-400a-9c78-52407e982478-tls-certs\") pod \"premium-w0-b21507-kserve-75db566f55-sxwvh\" (UID: \"38428a53-04c7-400a-9c78-52407e982478\") " pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh"
Oct 30 04:57:52.276857 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:52.276832    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrmnz\" (UniqueName: \"kubernetes.io/projected/38428a53-04c7-400a-9c78-52407e982478-kube-api-access-zrmnz\") pod \"premium-w0-b21507-kserve-75db566f55-sxwvh\" (UID: \"38428a53-04c7-400a-9c78-52407e982478\") " pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh"
Oct 30 04:57:52.304338 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:52.304313    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh"
Oct 30 04:57:52.452693 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:52.452661    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb"
Oct 30 04:57:52.464971 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:52.464949    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb"
Oct 30 04:57:52.597455 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:52.597413    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh"]
Oct 30 04:57:52.598940 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:57:52.598867    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38428a53_04c7_400a_9c78_52407e982478.slice/crio-9e4c9195c681f79150497445f821d0afe53ef111821571f73fc0d1f612ad1f63 WatchSource:0}: Error finding container 9e4c9195c681f79150497445f821d0afe53ef111821571f73fc0d1f612ad1f63: Status 404 returned error can't find the container with id 9e4c9195c681f79150497445f821d0afe53ef111821571f73fc0d1f612ad1f63
Oct 30 04:57:52.891028 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:52.890956    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:57:53.462302 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:53.462263    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh" event={"ID":"38428a53-04c7-400a-9c78-52407e982478","Type":"ContainerStarted","Data":"b702a116520d68edde52842852fcb463f2966912049c96ba49d8814ee70099e0"}
Oct 30 04:57:53.462761 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:53.462318    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh" event={"ID":"38428a53-04c7-400a-9c78-52407e982478","Type":"ContainerStarted","Data":"9e4c9195c681f79150497445f821d0afe53ef111821571f73fc0d1f612ad1f63"}
Oct 30 04:57:53.464856 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:53.464832    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb"
Oct 30 04:57:53.500246 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:53.500177    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh" podStartSLOduration=2.500161069 podStartE2EDuration="2.500161069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:57:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:57:53.498679207 +0000 UTC m=+802.722499109" watchObservedRunningTime="2026-10-30 04:57:53.500161069 +0000 UTC m=+802.723980972"
Oct 30 04:57:54.305416 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:54.305386    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh"
Oct 30 04:57:54.668307 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:54.668271    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:57:56.098932 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:56.098883    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 04:57:56.565078 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:56.564992    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:57:56.817058 ip-10-0-135-98 kubenswrapper[5898]: E1030 04:57:56.816969    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 04:57:58.201491 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:58.201373    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849"]
Oct 30 04:57:58.205449 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:58.205422    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849"
Oct 30 04:57:58.220574 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:58.220553    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-0b21ba\"/\"kube-root-ca.crt\""
Oct 30 04:57:58.222761 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:58.222735    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849"]
Oct 30 04:57:58.232217 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:58.232193    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-0b21ba\"/\"openshift-service-ca.crt\""
Oct 30 04:57:58.232563 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:58.232543    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-0b21ba\"/\"premium-w5-0b21ba-kserve-self-signed-certs\""
Oct 30 04:57:58.232648 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:58.232572    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-0b21ba\"/\"default-dockercfg-qgzhb\""
Oct 30 04:57:58.240943 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:58.240895    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:57:58.303326 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:58.303299    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9707f50-d5e1-409f-b8f3-c58629677aaf-tls-certs\") pod \"premium-w5-0b21ba-kserve-c8888d867-9z849\" (UID: \"a9707f50-d5e1-409f-b8f3-c58629677aaf\") " pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849"
Oct 30 04:57:58.303451 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:58.303332    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9707f50-d5e1-409f-b8f3-c58629677aaf-model-cache\") pod \"premium-w5-0b21ba-kserve-c8888d867-9z849\" (UID: \"a9707f50-d5e1-409f-b8f3-c58629677aaf\") " pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849"
Oct 30 04:57:58.303451 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:58.303359    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9707f50-d5e1-409f-b8f3-c58629677aaf-home\") pod \"premium-w5-0b21ba-kserve-c8888d867-9z849\" (UID: \"a9707f50-d5e1-409f-b8f3-c58629677aaf\") " pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849"
Oct 30 04:57:58.303574 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:58.303544    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9707f50-d5e1-409f-b8f3-c58629677aaf-dshm\") pod \"premium-w5-0b21ba-kserve-c8888d867-9z849\" (UID: \"a9707f50-d5e1-409f-b8f3-c58629677aaf\") " pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849"
Oct 30 04:57:58.303707 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:58.303649    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvbr9\" (UniqueName: \"kubernetes.io/projected/a9707f50-d5e1-409f-b8f3-c58629677aaf-kube-api-access-jvbr9\") pod \"premium-w5-0b21ba-kserve-c8888d867-9z849\" (UID: \"a9707f50-d5e1-409f-b8f3-c58629677aaf\") " pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849"
Oct 30 04:57:58.303707 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:58.303682    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9707f50-d5e1-409f-b8f3-c58629677aaf-tmp-dir\") pod \"premium-w5-0b21ba-kserve-c8888d867-9z849\" (UID: \"a9707f50-d5e1-409f-b8f3-c58629677aaf\") " pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849"
Oct 30 04:57:58.326075 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:58.326050    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:57:58.419917 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:58.419868    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9707f50-d5e1-409f-b8f3-c58629677aaf-dshm\") pod \"premium-w5-0b21ba-kserve-c8888d867-9z849\" (UID: \"a9707f50-d5e1-409f-b8f3-c58629677aaf\") " pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849"
Oct 30 04:57:58.420097 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:58.419963    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvbr9\" (UniqueName: \"kubernetes.io/projected/a9707f50-d5e1-409f-b8f3-c58629677aaf-kube-api-access-jvbr9\") pod \"premium-w5-0b21ba-kserve-c8888d867-9z849\" (UID: \"a9707f50-d5e1-409f-b8f3-c58629677aaf\") " pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849"
Oct 30 04:57:58.420097 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:58.420000    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9707f50-d5e1-409f-b8f3-c58629677aaf-tmp-dir\") pod \"premium-w5-0b21ba-kserve-c8888d867-9z849\" (UID: \"a9707f50-d5e1-409f-b8f3-c58629677aaf\") " pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849"
Oct 30 04:57:58.420097 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:58.420064    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9707f50-d5e1-409f-b8f3-c58629677aaf-tls-certs\") pod \"premium-w5-0b21ba-kserve-c8888d867-9z849\" (UID: \"a9707f50-d5e1-409f-b8f3-c58629677aaf\") " pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849"
Oct 30 04:57:58.420281 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:58.420095    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9707f50-d5e1-409f-b8f3-c58629677aaf-model-cache\") pod \"premium-w5-0b21ba-kserve-c8888d867-9z849\" (UID: \"a9707f50-d5e1-409f-b8f3-c58629677aaf\") " pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849"
Oct 30 04:57:58.420281 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:58.420133    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9707f50-d5e1-409f-b8f3-c58629677aaf-home\") pod \"premium-w5-0b21ba-kserve-c8888d867-9z849\" (UID: \"a9707f50-d5e1-409f-b8f3-c58629677aaf\") " pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849"
Oct 30 04:57:58.420601 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:58.420575    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9707f50-d5e1-409f-b8f3-c58629677aaf-home\") pod \"premium-w5-0b21ba-kserve-c8888d867-9z849\" (UID: \"a9707f50-d5e1-409f-b8f3-c58629677aaf\") " pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849"
Oct 30 04:57:58.423140 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:58.423108    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9707f50-d5e1-409f-b8f3-c58629677aaf-tmp-dir\") pod \"premium-w5-0b21ba-kserve-c8888d867-9z849\" (UID: \"a9707f50-d5e1-409f-b8f3-c58629677aaf\") " pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849"
Oct 30 04:57:58.423281 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:58.423085    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9707f50-d5e1-409f-b8f3-c58629677aaf-model-cache\") pod \"premium-w5-0b21ba-kserve-c8888d867-9z849\" (UID: \"a9707f50-d5e1-409f-b8f3-c58629677aaf\") " pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849"
Oct 30 04:57:58.424146 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:58.424118    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9707f50-d5e1-409f-b8f3-c58629677aaf-dshm\") pod \"premium-w5-0b21ba-kserve-c8888d867-9z849\" (UID: \"a9707f50-d5e1-409f-b8f3-c58629677aaf\") " pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849"
Oct 30 04:57:58.426456 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:58.426427    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9707f50-d5e1-409f-b8f3-c58629677aaf-tls-certs\") pod \"premium-w5-0b21ba-kserve-c8888d867-9z849\" (UID: \"a9707f50-d5e1-409f-b8f3-c58629677aaf\") " pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849"
Oct 30 04:57:58.441515 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:58.441489    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvbr9\" (UniqueName: \"kubernetes.io/projected/a9707f50-d5e1-409f-b8f3-c58629677aaf-kube-api-access-jvbr9\") pod \"premium-w5-0b21ba-kserve-c8888d867-9z849\" (UID: \"a9707f50-d5e1-409f-b8f3-c58629677aaf\") " pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849"
Oct 30 04:57:58.518446 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:58.518364    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849"
Oct 30 04:57:58.814227 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:58.814167    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849"]
Oct 30 04:57:58.816978 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:57:58.816951    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda9707f50_d5e1_409f_b8f3_c58629677aaf.slice/crio-5e746f6592babe76365d865b5926390fc3734cbd453092ddb115296418c46392 WatchSource:0}: Error finding container 5e746f6592babe76365d865b5926390fc3734cbd453092ddb115296418c46392: Status 404 returned error can't find the container with id 5e746f6592babe76365d865b5926390fc3734cbd453092ddb115296418c46392
Oct 30 04:57:59.496515 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:59.496477    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849" event={"ID":"a9707f50-d5e1-409f-b8f3-c58629677aaf","Type":"ContainerStarted","Data":"7847063f47a43c40439f8270962e51a01a8a90739899dc81fcdc3bf48373768a"}
Oct 30 04:57:59.496917 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:59.496523    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849" event={"ID":"a9707f50-d5e1-409f-b8f3-c58629677aaf","Type":"ContainerStarted","Data":"5e746f6592babe76365d865b5926390fc3734cbd453092ddb115296418c46392"}
Oct 30 04:57:59.519128 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:59.519098    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849"
Oct 30 04:57:59.536606 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:57:59.536570    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849" podStartSLOduration=1.536558931 podStartE2EDuration="1.536558931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:57:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:57:59.536131465 +0000 UTC m=+808.759951366" watchObservedRunningTime="2026-10-30 04:57:59.536558931 +0000 UTC m=+808.760378831"
Oct 30 04:58:00.014305 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:00.014270    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:58:01.756843 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:01.756809    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:58:01.943987 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:01.943945    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247"]
Oct 30 04:58:01.949249 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:01.949221    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247"
Oct 30 04:58:01.967391 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:01.967361    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247"]
Oct 30 04:58:01.976831 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:01.976804    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-889ca9\"/\"kube-root-ca.crt\""
Oct 30 04:58:01.977054 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:01.976862    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-889ca9\"/\"premium-w3-889ca9-kserve-self-signed-certs\""
Oct 30 04:58:01.977152 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:01.976885    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-889ca9\"/\"default-dockercfg-gf5t6\""
Oct 30 04:58:01.977152 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:01.976946    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-889ca9\"/\"openshift-service-ca.crt\""
Oct 30 04:58:02.003830 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:02.003798    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:58:02.053421 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:02.053332    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d865d3e-a6e9-407e-844e-cc75fc7f1354-dshm\") pod \"premium-w3-889ca9-kserve-57dd9bb5fc-ql247\" (UID: \"1d865d3e-a6e9-407e-844e-cc75fc7f1354\") " pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247"
Oct 30 04:58:02.053421 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:02.053395    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d865d3e-a6e9-407e-844e-cc75fc7f1354-home\") pod \"premium-w3-889ca9-kserve-57dd9bb5fc-ql247\" (UID: \"1d865d3e-a6e9-407e-844e-cc75fc7f1354\") " pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247"
Oct 30 04:58:02.053653 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:02.053493    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d865d3e-a6e9-407e-844e-cc75fc7f1354-tmp-dir\") pod \"premium-w3-889ca9-kserve-57dd9bb5fc-ql247\" (UID: \"1d865d3e-a6e9-407e-844e-cc75fc7f1354\") " pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247"
Oct 30 04:58:02.053653 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:02.053518    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d865d3e-a6e9-407e-844e-cc75fc7f1354-model-cache\") pod \"premium-w3-889ca9-kserve-57dd9bb5fc-ql247\" (UID: \"1d865d3e-a6e9-407e-844e-cc75fc7f1354\") " pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247"
Oct 30 04:58:02.053763 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:02.053688    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d865d3e-a6e9-407e-844e-cc75fc7f1354-tls-certs\") pod \"premium-w3-889ca9-kserve-57dd9bb5fc-ql247\" (UID: \"1d865d3e-a6e9-407e-844e-cc75fc7f1354\") " pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247"
Oct 30 04:58:02.053763 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:02.053726    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhp8f\" (UniqueName: \"kubernetes.io/projected/1d865d3e-a6e9-407e-844e-cc75fc7f1354-kube-api-access-nhp8f\") pod \"premium-w3-889ca9-kserve-57dd9bb5fc-ql247\" (UID: \"1d865d3e-a6e9-407e-844e-cc75fc7f1354\") " pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247"
Oct 30 04:58:02.155103 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:02.155060    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhp8f\" (UniqueName: \"kubernetes.io/projected/1d865d3e-a6e9-407e-844e-cc75fc7f1354-kube-api-access-nhp8f\") pod \"premium-w3-889ca9-kserve-57dd9bb5fc-ql247\" (UID: \"1d865d3e-a6e9-407e-844e-cc75fc7f1354\") " pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247"
Oct 30 04:58:02.155320 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:02.155137    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d865d3e-a6e9-407e-844e-cc75fc7f1354-dshm\") pod \"premium-w3-889ca9-kserve-57dd9bb5fc-ql247\" (UID: \"1d865d3e-a6e9-407e-844e-cc75fc7f1354\") " pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247"
Oct 30 04:58:02.155320 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:02.155173    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d865d3e-a6e9-407e-844e-cc75fc7f1354-home\") pod \"premium-w3-889ca9-kserve-57dd9bb5fc-ql247\" (UID: \"1d865d3e-a6e9-407e-844e-cc75fc7f1354\") " pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247"
Oct 30 04:58:02.155320 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:02.155210    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d865d3e-a6e9-407e-844e-cc75fc7f1354-tmp-dir\") pod \"premium-w3-889ca9-kserve-57dd9bb5fc-ql247\" (UID: \"1d865d3e-a6e9-407e-844e-cc75fc7f1354\") " pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247"
Oct 30 04:58:02.155320 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:02.155238    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d865d3e-a6e9-407e-844e-cc75fc7f1354-model-cache\") pod \"premium-w3-889ca9-kserve-57dd9bb5fc-ql247\" (UID: \"1d865d3e-a6e9-407e-844e-cc75fc7f1354\") " pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247"
Oct 30 04:58:02.155540 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:02.155354    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d865d3e-a6e9-407e-844e-cc75fc7f1354-tls-certs\") pod \"premium-w3-889ca9-kserve-57dd9bb5fc-ql247\" (UID: \"1d865d3e-a6e9-407e-844e-cc75fc7f1354\") " pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247"
Oct 30 04:58:02.156327 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:02.156298    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d865d3e-a6e9-407e-844e-cc75fc7f1354-tmp-dir\") pod \"premium-w3-889ca9-kserve-57dd9bb5fc-ql247\" (UID: \"1d865d3e-a6e9-407e-844e-cc75fc7f1354\") " pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247"
Oct 30 04:58:02.156741 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:02.156718    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d865d3e-a6e9-407e-844e-cc75fc7f1354-model-cache\") pod \"premium-w3-889ca9-kserve-57dd9bb5fc-ql247\" (UID: \"1d865d3e-a6e9-407e-844e-cc75fc7f1354\") " pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247"
Oct 30 04:58:02.157215 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:02.157194    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d865d3e-a6e9-407e-844e-cc75fc7f1354-home\") pod \"premium-w3-889ca9-kserve-57dd9bb5fc-ql247\" (UID: \"1d865d3e-a6e9-407e-844e-cc75fc7f1354\") " pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247"
Oct 30 04:58:02.164696 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:02.164675    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d865d3e-a6e9-407e-844e-cc75fc7f1354-tls-certs\") pod \"premium-w3-889ca9-kserve-57dd9bb5fc-ql247\" (UID: \"1d865d3e-a6e9-407e-844e-cc75fc7f1354\") " pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247"
Oct 30 04:58:02.166147 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:02.166119    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d865d3e-a6e9-407e-844e-cc75fc7f1354-dshm\") pod \"premium-w3-889ca9-kserve-57dd9bb5fc-ql247\" (UID: \"1d865d3e-a6e9-407e-844e-cc75fc7f1354\") " pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247"
Oct 30 04:58:02.174142 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:02.174117    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhp8f\" (UniqueName: \"kubernetes.io/projected/1d865d3e-a6e9-407e-844e-cc75fc7f1354-kube-api-access-nhp8f\") pod \"premium-w3-889ca9-kserve-57dd9bb5fc-ql247\" (UID: \"1d865d3e-a6e9-407e-844e-cc75fc7f1354\") " pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247"
Oct 30 04:58:02.265413 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:02.265376    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247"
Oct 30 04:58:02.305242 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:02.305170    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh"
Oct 30 04:58:02.318758 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:02.318730    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh"
Oct 30 04:58:02.524915 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:02.524878    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh"
Oct 30 04:58:02.569644 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:02.569615    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247"]
Oct 30 04:58:02.572005 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:58:02.571983    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d865d3e_a6e9_407e_844e_cc75fc7f1354.slice/crio-4f0b238e0e5670439e0b22c2caa00ac0a5759645260914360adb395ea1fd376c WatchSource:0}: Error finding container 4f0b238e0e5670439e0b22c2caa00ac0a5759645260914360adb395ea1fd376c: Status 404 returned error can't find the container with id 4f0b238e0e5670439e0b22c2caa00ac0a5759645260914360adb395ea1fd376c
Oct 30 04:58:03.472715 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:03.472682    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:58:03.520611 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:03.520576    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247" event={"ID":"1d865d3e-a6e9-407e-844e-cc75fc7f1354","Type":"ContainerStarted","Data":"184b8e65fb984b00278e69cb1a81946da12ac9ff7d0733cfb1bcc254412d30c8"}
Oct 30 04:58:03.520783 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:03.520621    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247" event={"ID":"1d865d3e-a6e9-407e-844e-cc75fc7f1354","Type":"ContainerStarted","Data":"4f0b238e0e5670439e0b22c2caa00ac0a5759645260914360adb395ea1fd376c"}
Oct 30 04:58:03.557017 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:03.556968    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247" podStartSLOduration=2.556954202 podStartE2EDuration="2.556954202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:58:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:58:03.556429235 +0000 UTC m=+812.780249133" watchObservedRunningTime="2026-10-30 04:58:03.556954202 +0000 UTC m=+812.780774102"
Oct 30 04:58:03.764399 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:03.764317    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:58:04.265812 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:04.265780    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247"
Oct 30 04:58:05.079447 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:05.079408    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:58:05.404282 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:05.404201    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:58:06.743706 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:06.743670    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:58:07.860243 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:07.860211    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5"]
Oct 30 04:58:07.864158 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:07.864135    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5"
Oct 30 04:58:07.883808 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:07.883783    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5"]
Oct 30 04:58:07.891973 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:07.891860    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-cff4ac\"/\"kube-root-ca.crt\""
Oct 30 04:58:07.891973 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:07.891867    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-cff4ac\"/\"default-dockercfg-7dcgl\""
Oct 30 04:58:07.891973 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:07.891910    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-cff4ac\"/\"premium-w1-cff4ac-kserve-self-signed-certs\""
Oct 30 04:58:07.891973 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:07.891962    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-cff4ac\"/\"openshift-service-ca.crt\""
Oct 30 04:58:07.900157 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:07.900133    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:58:07.911345 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:07.911317    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebf8f4a2-8ac6-48ec-bdba-3f6a99653264-tls-certs\") pod \"premium-w1-cff4ac-kserve-7b4db76fbd-75cn5\" (UID: \"ebf8f4a2-8ac6-48ec-bdba-3f6a99653264\") " pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5"
Oct 30 04:58:07.911449 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:07.911365    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-255jw\" (UniqueName: \"kubernetes.io/projected/ebf8f4a2-8ac6-48ec-bdba-3f6a99653264-kube-api-access-255jw\") pod \"premium-w1-cff4ac-kserve-7b4db76fbd-75cn5\" (UID: \"ebf8f4a2-8ac6-48ec-bdba-3f6a99653264\") " pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5"
Oct 30 04:58:07.911494 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:07.911479    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebf8f4a2-8ac6-48ec-bdba-3f6a99653264-tmp-dir\") pod \"premium-w1-cff4ac-kserve-7b4db76fbd-75cn5\" (UID: \"ebf8f4a2-8ac6-48ec-bdba-3f6a99653264\") " pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5"
Oct 30 04:58:07.911602 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:07.911581    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebf8f4a2-8ac6-48ec-bdba-3f6a99653264-dshm\") pod \"premium-w1-cff4ac-kserve-7b4db76fbd-75cn5\" (UID: \"ebf8f4a2-8ac6-48ec-bdba-3f6a99653264\") " pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5"
Oct 30 04:58:07.911664 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:07.911626    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebf8f4a2-8ac6-48ec-bdba-3f6a99653264-model-cache\") pod \"premium-w1-cff4ac-kserve-7b4db76fbd-75cn5\" (UID: \"ebf8f4a2-8ac6-48ec-bdba-3f6a99653264\") " pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5"
Oct 30 04:58:07.911710 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:07.911694    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebf8f4a2-8ac6-48ec-bdba-3f6a99653264-home\") pod \"premium-w1-cff4ac-kserve-7b4db76fbd-75cn5\" (UID: \"ebf8f4a2-8ac6-48ec-bdba-3f6a99653264\") " pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5"
Oct 30 04:58:08.012270 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:08.012234    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebf8f4a2-8ac6-48ec-bdba-3f6a99653264-tls-certs\") pod \"premium-w1-cff4ac-kserve-7b4db76fbd-75cn5\" (UID: \"ebf8f4a2-8ac6-48ec-bdba-3f6a99653264\") " pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5"
Oct 30 04:58:08.012270 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:08.012273    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-255jw\" (UniqueName: \"kubernetes.io/projected/ebf8f4a2-8ac6-48ec-bdba-3f6a99653264-kube-api-access-255jw\") pod \"premium-w1-cff4ac-kserve-7b4db76fbd-75cn5\" (UID: \"ebf8f4a2-8ac6-48ec-bdba-3f6a99653264\") " pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5"
Oct 30 04:58:08.012528 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:08.012301    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebf8f4a2-8ac6-48ec-bdba-3f6a99653264-tmp-dir\") pod \"premium-w1-cff4ac-kserve-7b4db76fbd-75cn5\" (UID: \"ebf8f4a2-8ac6-48ec-bdba-3f6a99653264\") " pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5"
Oct 30 04:58:08.012528 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:08.012348    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebf8f4a2-8ac6-48ec-bdba-3f6a99653264-dshm\") pod \"premium-w1-cff4ac-kserve-7b4db76fbd-75cn5\" (UID: \"ebf8f4a2-8ac6-48ec-bdba-3f6a99653264\") " pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5"
Oct 30 04:58:08.012528 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:08.012367    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebf8f4a2-8ac6-48ec-bdba-3f6a99653264-model-cache\") pod \"premium-w1-cff4ac-kserve-7b4db76fbd-75cn5\" (UID: \"ebf8f4a2-8ac6-48ec-bdba-3f6a99653264\") " pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5"
Oct 30 04:58:08.012528 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:08.012403    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebf8f4a2-8ac6-48ec-bdba-3f6a99653264-home\") pod \"premium-w1-cff4ac-kserve-7b4db76fbd-75cn5\" (UID: \"ebf8f4a2-8ac6-48ec-bdba-3f6a99653264\") " pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5"
Oct 30 04:58:08.012960 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:08.012932    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebf8f4a2-8ac6-48ec-bdba-3f6a99653264-tmp-dir\") pod \"premium-w1-cff4ac-kserve-7b4db76fbd-75cn5\" (UID: \"ebf8f4a2-8ac6-48ec-bdba-3f6a99653264\") " pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5"
Oct 30 04:58:08.013105 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:08.012962    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebf8f4a2-8ac6-48ec-bdba-3f6a99653264-model-cache\") pod \"premium-w1-cff4ac-kserve-7b4db76fbd-75cn5\" (UID: \"ebf8f4a2-8ac6-48ec-bdba-3f6a99653264\") " pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5"
Oct 30 04:58:08.013105 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:08.012986    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebf8f4a2-8ac6-48ec-bdba-3f6a99653264-home\") pod \"premium-w1-cff4ac-kserve-7b4db76fbd-75cn5\" (UID: \"ebf8f4a2-8ac6-48ec-bdba-3f6a99653264\") " pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5"
Oct 30 04:58:08.014856 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:08.014835    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebf8f4a2-8ac6-48ec-bdba-3f6a99653264-tls-certs\") pod \"premium-w1-cff4ac-kserve-7b4db76fbd-75cn5\" (UID: \"ebf8f4a2-8ac6-48ec-bdba-3f6a99653264\") " pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5"
Oct 30 04:58:08.015055 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:08.015035    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebf8f4a2-8ac6-48ec-bdba-3f6a99653264-dshm\") pod \"premium-w1-cff4ac-kserve-7b4db76fbd-75cn5\" (UID: \"ebf8f4a2-8ac6-48ec-bdba-3f6a99653264\") " pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5"
Oct 30 04:58:08.030463 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:08.030441    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-255jw\" (UniqueName: \"kubernetes.io/projected/ebf8f4a2-8ac6-48ec-bdba-3f6a99653264-kube-api-access-255jw\") pod \"premium-w1-cff4ac-kserve-7b4db76fbd-75cn5\" (UID: \"ebf8f4a2-8ac6-48ec-bdba-3f6a99653264\") " pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5"
Oct 30 04:58:08.201422 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:08.201340    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5"
Oct 30 04:58:08.375888 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:08.375859    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:58:08.496047 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:08.496010    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5"]
Oct 30 04:58:08.497810 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:58:08.497784    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podebf8f4a2_8ac6_48ec_bdba_3f6a99653264.slice/crio-ca04a46dedd10ccfc75e70bc1b94b6e53db786d7763db20d400e9eec6cc89149 WatchSource:0}: Error finding container ca04a46dedd10ccfc75e70bc1b94b6e53db786d7763db20d400e9eec6cc89149: Status 404 returned error can't find the container with id ca04a46dedd10ccfc75e70bc1b94b6e53db786d7763db20d400e9eec6cc89149
Oct 30 04:58:08.518542 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:08.518519    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849"
Oct 30 04:58:08.531870 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:08.531852    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849"
Oct 30 04:58:08.547187 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:08.547160    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5" event={"ID":"ebf8f4a2-8ac6-48ec-bdba-3f6a99653264","Type":"ContainerStarted","Data":"ca04a46dedd10ccfc75e70bc1b94b6e53db786d7763db20d400e9eec6cc89149"}
Oct 30 04:58:08.558866 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:08.558845    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849"
Oct 30 04:58:09.554262 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:09.554216    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5" event={"ID":"ebf8f4a2-8ac6-48ec-bdba-3f6a99653264","Type":"ContainerStarted","Data":"56cd3db553627014ce617bcbdab0f5dc61d13d2be7b18028f5c557283fc8e56b"}
Oct 30 04:58:09.592641 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:09.592593    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5" podStartSLOduration=2.592578859 podStartE2EDuration="2.592578859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:58:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:58:09.591166902 +0000 UTC m=+818.814986822" watchObservedRunningTime="2026-10-30 04:58:09.592578859 +0000 UTC m=+818.816398762"
Oct 30 04:58:10.104010 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:10.103977    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:58:10.202301 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:10.202271    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5"
Oct 30 04:58:10.216783 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:10.216754    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 04:58:12.115886 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:12.115803    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:58:12.265770 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:12.265731    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247"
Oct 30 04:58:12.279185 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:12.279152    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247"
Oct 30 04:58:12.586775 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:12.586750    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247"
Oct 30 04:58:13.716777 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:13.716737    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:58:15.403038 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:15.403003    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:58:17.130310 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:17.130273    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:58:18.201509 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:18.201477    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5"
Oct 30 04:58:18.214016 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:18.213986    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5"
Oct 30 04:58:18.616548 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:18.616513    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5"
Oct 30 04:58:18.841974 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:18.841938    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:58:20.408281 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:20.408244    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:58:22.293688 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:22.293656    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:58:24.153771 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:24.153736    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:58:26.000942 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:26.000890    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:58:26.818285 ip-10-0-135-98 kubenswrapper[5898]: E1030 04:58:26.818250    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 04:58:27.839633 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:27.839595    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:58:29.378998 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:29.378963    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00fda0ea-b5tnk"]
Oct 30 04:58:29.382712 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:29.382691    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00fda0ea-b5tnk"
Oct 30 04:58:29.397149 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:29.397128    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:58:29.408459 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:29.408431    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\""
Oct 30 04:58:29.410644 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:29.410620    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00fda0ea-b5tnk"]
Oct 30 04:58:29.517574 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:29.517545    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqqbw\" (UniqueName: \"kubernetes.io/projected/01f80893-ab5c-41c6-8f8d-4e578bfb0feb-kube-api-access-hqqbw\") pod \"maas-api-revoke-sub-00fda0ea-b5tnk\" (UID: \"01f80893-ab5c-41c6-8f8d-4e578bfb0feb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00fda0ea-b5tnk"
Oct 30 04:58:29.517762 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:29.517598    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01f80893-ab5c-41c6-8f8d-4e578bfb0feb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00fda0ea-b5tnk\" (UID: \"01f80893-ab5c-41c6-8f8d-4e578bfb0feb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00fda0ea-b5tnk"
Oct 30 04:58:29.517762 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:29.517667    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01f80893-ab5c-41c6-8f8d-4e578bfb0feb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00fda0ea-b5tnk\" (UID: \"01f80893-ab5c-41c6-8f8d-4e578bfb0feb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00fda0ea-b5tnk"
Oct 30 04:58:29.545856 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:29.545822    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:58:29.619419 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:29.619389    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01f80893-ab5c-41c6-8f8d-4e578bfb0feb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00fda0ea-b5tnk\" (UID: \"01f80893-ab5c-41c6-8f8d-4e578bfb0feb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00fda0ea-b5tnk"
Oct 30 04:58:29.619618 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:29.619510    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqqbw\" (UniqueName: \"kubernetes.io/projected/01f80893-ab5c-41c6-8f8d-4e578bfb0feb-kube-api-access-hqqbw\") pod \"maas-api-revoke-sub-00fda0ea-b5tnk\" (UID: \"01f80893-ab5c-41c6-8f8d-4e578bfb0feb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00fda0ea-b5tnk"
Oct 30 04:58:29.619618 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:29.619567    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01f80893-ab5c-41c6-8f8d-4e578bfb0feb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00fda0ea-b5tnk\" (UID: \"01f80893-ab5c-41c6-8f8d-4e578bfb0feb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00fda0ea-b5tnk"
Oct 30 04:58:29.620249 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:29.620223    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01f80893-ab5c-41c6-8f8d-4e578bfb0feb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00fda0ea-b5tnk\" (UID: \"01f80893-ab5c-41c6-8f8d-4e578bfb0feb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00fda0ea-b5tnk"
Oct 30 04:58:29.622274 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:29.622245    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01f80893-ab5c-41c6-8f8d-4e578bfb0feb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00fda0ea-b5tnk\" (UID: \"01f80893-ab5c-41c6-8f8d-4e578bfb0feb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00fda0ea-b5tnk"
Oct 30 04:58:29.637561 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:29.637506    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqqbw\" (UniqueName: \"kubernetes.io/projected/01f80893-ab5c-41c6-8f8d-4e578bfb0feb-kube-api-access-hqqbw\") pod \"maas-api-revoke-sub-00fda0ea-b5tnk\" (UID: \"01f80893-ab5c-41c6-8f8d-4e578bfb0feb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00fda0ea-b5tnk"
Oct 30 04:58:29.698461 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:29.698435    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00fda0ea-b5tnk"
Oct 30 04:58:29.995668 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:29.994778    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00fda0ea-b5tnk"]
Oct 30 04:58:29.997662 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:58:29.997635    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01f80893_ab5c_41c6_8f8d_4e578bfb0feb.slice/crio-035bdc4db0213b0cd3f828a22a1c721e5332c63f49f2f29f21dec35bfda04f31 WatchSource:0}: Error finding container 035bdc4db0213b0cd3f828a22a1c721e5332c63f49f2f29f21dec35bfda04f31: Status 404 returned error can't find the container with id 035bdc4db0213b0cd3f828a22a1c721e5332c63f49f2f29f21dec35bfda04f31
Oct 30 04:58:30.671284 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:30.671249    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00fda0ea-b5tnk" event={"ID":"01f80893-ab5c-41c6-8f8d-4e578bfb0feb","Type":"ContainerStarted","Data":"035bdc4db0213b0cd3f828a22a1c721e5332c63f49f2f29f21dec35bfda04f31"}
Oct 30 04:58:31.444741 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:31.444703    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:58:32.684153 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:32.684077    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00fda0ea-b5tnk" event={"ID":"01f80893-ab5c-41c6-8f8d-4e578bfb0feb","Type":"ContainerStarted","Data":"4ad7148950bb3ce811c6aee370940875df20b448b3a3064e6e4c1ba75cfa057e"}
Oct 30 04:58:32.721689 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:32.721641    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00fda0ea-b5tnk" podStartSLOduration=1.5930702669999999 podStartE2EDuration="3.721627473s" totalImagesPullingTime="2.128557206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:58:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:58:32.720110969 +0000 UTC m=+841.943930871" watchObservedRunningTime="2026-10-30 04:58:32.721627473 +0000 UTC m=+841.945447373"
Oct 30 04:58:33.108888 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:33.108853    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 04:58:33.118497 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:33.118473    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:58:33.690918 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:33.690869    5898 generic.go:356] "Generic (PLEG): container finished" podID="01f80893-ab5c-41c6-8f8d-4e578bfb0feb" containerID="4ad7148950bb3ce811c6aee370940875df20b448b3a3064e6e4c1ba75cfa057e" exitCode=0
Oct 30 04:58:33.691418 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:33.690947    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00fda0ea-b5tnk" event={"ID":"01f80893-ab5c-41c6-8f8d-4e578bfb0feb","Type":"ContainerDied","Data":"4ad7148950bb3ce811c6aee370940875df20b448b3a3064e6e4c1ba75cfa057e"}
Oct 30 04:58:34.837030 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:34.837005    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00fda0ea-b5tnk"
Oct 30 04:58:34.866883 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:34.866859    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01f80893-ab5c-41c6-8f8d-4e578bfb0feb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01f80893-ab5c-41c6-8f8d-4e578bfb0feb-maas-api-openshift-ca\") pod \"01f80893-ab5c-41c6-8f8d-4e578bfb0feb\" (UID: \"01f80893-ab5c-41c6-8f8d-4e578bfb0feb\") "
Oct 30 04:58:34.867056 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:34.866917    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01f80893-ab5c-41c6-8f8d-4e578bfb0feb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01f80893-ab5c-41c6-8f8d-4e578bfb0feb-maas-api-xks-ca\") pod \"01f80893-ab5c-41c6-8f8d-4e578bfb0feb\" (UID: \"01f80893-ab5c-41c6-8f8d-4e578bfb0feb\") "
Oct 30 04:58:34.867056 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:34.866992    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01f80893-ab5c-41c6-8f8d-4e578bfb0feb-kube-api-access-hqqbw\" (UniqueName: \"kubernetes.io/projected/01f80893-ab5c-41c6-8f8d-4e578bfb0feb-kube-api-access-hqqbw\") pod \"01f80893-ab5c-41c6-8f8d-4e578bfb0feb\" (UID: \"01f80893-ab5c-41c6-8f8d-4e578bfb0feb\") "
Oct 30 04:58:34.867301 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:34.867270    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01f80893-ab5c-41c6-8f8d-4e578bfb0feb-maas-api-openshift-ca" pod "01f80893-ab5c-41c6-8f8d-4e578bfb0feb" (UID: "01f80893-ab5c-41c6-8f8d-4e578bfb0feb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:58:34.869229 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:34.869203    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01f80893-ab5c-41c6-8f8d-4e578bfb0feb-maas-api-xks-ca" pod "01f80893-ab5c-41c6-8f8d-4e578bfb0feb" (UID: "01f80893-ab5c-41c6-8f8d-4e578bfb0feb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:58:34.869319 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:34.869295    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01f80893-ab5c-41c6-8f8d-4e578bfb0feb-kube-api-access-hqqbw" pod "01f80893-ab5c-41c6-8f8d-4e578bfb0feb" (UID: "01f80893-ab5c-41c6-8f8d-4e578bfb0feb"). InnerVolumeSpecName "kube-api-access-hqqbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:58:34.968594 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:34.968521    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01f80893-ab5c-41c6-8f8d-4e578bfb0feb-maas-api-openshift-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 04:58:34.968594 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:34.968551    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01f80893-ab5c-41c6-8f8d-4e578bfb0feb-maas-api-xks-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 04:58:34.968594 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:34.968561    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqqbw\" (UniqueName: \"kubernetes.io/projected/01f80893-ab5c-41c6-8f8d-4e578bfb0feb-kube-api-access-hqqbw\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 04:58:35.075195 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:35.075152    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:58:35.705194 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:35.705167    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00fda0ea-b5tnk"
Oct 30 04:58:35.705365 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:35.705189    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00fda0ea-b5tnk" event={"ID":"01f80893-ab5c-41c6-8f8d-4e578bfb0feb","Type":"ContainerDied","Data":"035bdc4db0213b0cd3f828a22a1c721e5332c63f49f2f29f21dec35bfda04f31"}
Oct 30 04:58:35.705365 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:35.705266    5898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="035bdc4db0213b0cd3f828a22a1c721e5332c63f49f2f29f21dec35bfda04f31"
Oct 30 04:58:36.394535 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:36.394501    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl"]
Oct 30 04:58:36.395031 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:36.394879    5898 cpu_manager.go:414] "containerMap: removing container" podUID="01f80893-ab5c-41c6-8f8d-4e578bfb0feb" containerName="revoke-keys"
Oct 30 04:58:36.395031 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:36.394893    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01f80893-ab5c-41c6-8f8d-4e578bfb0feb" containerName="revoke-keys"
Oct 30 04:58:36.395031 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:36.395006    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="01f80893-ab5c-41c6-8f8d-4e578bfb0feb" containerName="revoke-keys"
Oct 30 04:58:36.398323 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:36.398301    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl"
Oct 30 04:58:36.422184 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:36.422157    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-83e83e-openshift-default-dockercfg-zgpn5\""
Oct 30 04:58:36.437326 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:36.437298    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl"]
Oct 30 04:58:36.481095 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:36.481064    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58083aef-e47e-4ba5-84f0-a70d80973181-workload-socket\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl\" (UID: \"58083aef-e47e-4ba5-84f0-a70d80973181\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl"
Oct 30 04:58:36.481235 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:36.481099    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58083aef-e47e-4ba5-84f0-a70d80973181-credential-socket\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl\" (UID: \"58083aef-e47e-4ba5-84f0-a70d80973181\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl"
Oct 30 04:58:36.481235 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:36.481131    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58083aef-e47e-4ba5-84f0-a70d80973181-istio-envoy\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl\" (UID: \"58083aef-e47e-4ba5-84f0-a70d80973181\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl"
Oct 30 04:58:36.481235 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:36.481222    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58083aef-e47e-4ba5-84f0-a70d80973181-workload-certs\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl\" (UID: \"58083aef-e47e-4ba5-84f0-a70d80973181\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl"
Oct 30 04:58:36.481344 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:36.481270    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58083aef-e47e-4ba5-84f0-a70d80973181-istio-token\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl\" (UID: \"58083aef-e47e-4ba5-84f0-a70d80973181\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl"
Oct 30 04:58:36.481376 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:36.481346    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58083aef-e47e-4ba5-84f0-a70d80973181-istio-podinfo\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl\" (UID: \"58083aef-e47e-4ba5-84f0-a70d80973181\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl"
Oct 30 04:58:36.481408 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:36.481398    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58083aef-e47e-4ba5-84f0-a70d80973181-istio-data\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl\" (UID: \"58083aef-e47e-4ba5-84f0-a70d80973181\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl"
Oct 30 04:58:36.481444 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:36.481437    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58083aef-e47e-4ba5-84f0-a70d80973181-istiod-ca-cert\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl\" (UID: \"58083aef-e47e-4ba5-84f0-a70d80973181\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl"
Oct 30 04:58:36.481490 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:36.481454    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtztk\" (UniqueName: \"kubernetes.io/projected/58083aef-e47e-4ba5-84f0-a70d80973181-kube-api-access-xtztk\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl\" (UID: \"58083aef-e47e-4ba5-84f0-a70d80973181\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl"
Oct 30 04:58:36.582030 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:36.581992    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58083aef-e47e-4ba5-84f0-a70d80973181-workload-socket\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl\" (UID: \"58083aef-e47e-4ba5-84f0-a70d80973181\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl"
Oct 30 04:58:36.582030 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:36.582027    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58083aef-e47e-4ba5-84f0-a70d80973181-credential-socket\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl\" (UID: \"58083aef-e47e-4ba5-84f0-a70d80973181\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl"
Oct 30 04:58:36.582261 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:36.582058    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58083aef-e47e-4ba5-84f0-a70d80973181-istio-envoy\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl\" (UID: \"58083aef-e47e-4ba5-84f0-a70d80973181\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl"
Oct 30 04:58:36.582261 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:36.582095    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58083aef-e47e-4ba5-84f0-a70d80973181-workload-certs\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl\" (UID: \"58083aef-e47e-4ba5-84f0-a70d80973181\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl"
Oct 30 04:58:36.582261 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:36.582140    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58083aef-e47e-4ba5-84f0-a70d80973181-istio-token\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl\" (UID: \"58083aef-e47e-4ba5-84f0-a70d80973181\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl"
Oct 30 04:58:36.582261 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:36.582164    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58083aef-e47e-4ba5-84f0-a70d80973181-istio-podinfo\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl\" (UID: \"58083aef-e47e-4ba5-84f0-a70d80973181\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl"
Oct 30 04:58:36.582261 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:36.582213    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58083aef-e47e-4ba5-84f0-a70d80973181-istio-data\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl\" (UID: \"58083aef-e47e-4ba5-84f0-a70d80973181\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl"
Oct 30 04:58:36.582509 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:36.582443    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58083aef-e47e-4ba5-84f0-a70d80973181-workload-socket\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl\" (UID: \"58083aef-e47e-4ba5-84f0-a70d80973181\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl"
Oct 30 04:58:36.582561 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:36.582542    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58083aef-e47e-4ba5-84f0-a70d80973181-credential-socket\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl\" (UID: \"58083aef-e47e-4ba5-84f0-a70d80973181\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl"
Oct 30 04:58:36.582644 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:36.582606    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58083aef-e47e-4ba5-84f0-a70d80973181-workload-certs\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl\" (UID: \"58083aef-e47e-4ba5-84f0-a70d80973181\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl"
Oct 30 04:58:36.582776 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:36.582655    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58083aef-e47e-4ba5-84f0-a70d80973181-istio-data\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl\" (UID: \"58083aef-e47e-4ba5-84f0-a70d80973181\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl"
Oct 30 04:58:36.582776 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:36.582660    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58083aef-e47e-4ba5-84f0-a70d80973181-istiod-ca-cert\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl\" (UID: \"58083aef-e47e-4ba5-84f0-a70d80973181\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl"
Oct 30 04:58:36.582776 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:36.582703    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtztk\" (UniqueName: \"kubernetes.io/projected/58083aef-e47e-4ba5-84f0-a70d80973181-kube-api-access-xtztk\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl\" (UID: \"58083aef-e47e-4ba5-84f0-a70d80973181\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl"
Oct 30 04:58:36.583259 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:36.583237    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58083aef-e47e-4ba5-84f0-a70d80973181-istiod-ca-cert\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl\" (UID: \"58083aef-e47e-4ba5-84f0-a70d80973181\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl"
Oct 30 04:58:36.584436 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:36.584411    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58083aef-e47e-4ba5-84f0-a70d80973181-istio-envoy\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl\" (UID: \"58083aef-e47e-4ba5-84f0-a70d80973181\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl"
Oct 30 04:58:36.584736 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:36.584716    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58083aef-e47e-4ba5-84f0-a70d80973181-istio-podinfo\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl\" (UID: \"58083aef-e47e-4ba5-84f0-a70d80973181\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl"
Oct 30 04:58:36.603036 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:36.603004    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58083aef-e47e-4ba5-84f0-a70d80973181-istio-token\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl\" (UID: \"58083aef-e47e-4ba5-84f0-a70d80973181\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl"
Oct 30 04:58:36.603229 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:36.603209    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtztk\" (UniqueName: \"kubernetes.io/projected/58083aef-e47e-4ba5-84f0-a70d80973181-kube-api-access-xtztk\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl\" (UID: \"58083aef-e47e-4ba5-84f0-a70d80973181\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl"
Oct 30 04:58:36.710945 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:36.710850    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl"
Oct 30 04:58:36.771003 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:36.768467    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:58:37.020100 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:37.020075    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl"]
Oct 30 04:58:37.022311 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:58:37.022280    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58083aef_e47e_4ba5_84f0_a70d80973181.slice/crio-d0cec78f9ef1eecb84315a0caf2fd27b673e9f1196d6e23e8c865e2ba8824124 WatchSource:0}: Error finding container d0cec78f9ef1eecb84315a0caf2fd27b673e9f1196d6e23e8c865e2ba8824124: Status 404 returned error can't find the container with id d0cec78f9ef1eecb84315a0caf2fd27b673e9f1196d6e23e8c865e2ba8824124
Oct 30 04:58:37.024637 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:37.024609    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 04:58:37.024701 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:37.024672    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 04:58:37.024740 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:37.024699    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 04:58:37.717511 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:37.717471    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl" event={"ID":"58083aef-e47e-4ba5-84f0-a70d80973181","Type":"ContainerStarted","Data":"c674cfaa68f9547135031751a24ff0d52cc962e6a7adf05a35e611623a887b76"}
Oct 30 04:58:37.717949 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:37.717538    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl" event={"ID":"58083aef-e47e-4ba5-84f0-a70d80973181","Type":"ContainerStarted","Data":"d0cec78f9ef1eecb84315a0caf2fd27b673e9f1196d6e23e8c865e2ba8824124"}
Oct 30 04:58:37.768147 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:37.768098    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl" podStartSLOduration=1.768085868 podStartE2EDuration="1.768085868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:58:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:58:37.767176805 +0000 UTC m=+846.990996739" watchObservedRunningTime="2026-10-30 04:58:37.768085868 +0000 UTC m=+846.991905770"
Oct 30 04:58:38.112872 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:38.112833    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:58:38.590924 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:38.589038    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:58:38.711939 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:38.711849    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl"
Oct 30 04:58:38.713402 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:38.713361    5898 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 04:58:38.713453 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:38.713426    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl" podUID="58083aef-e47e-4ba5-84f0-a70d80973181" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 04:58:39.711957 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:39.711922    5898 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 04:58:39.712361 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:39.711996    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl" podUID="58083aef-e47e-4ba5-84f0-a70d80973181" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 04:58:40.322952 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:40.322921    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:58:40.711472 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:40.711389    5898 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 04:58:40.711472 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:40.711441    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl" podUID="58083aef-e47e-4ba5-84f0-a70d80973181" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 04:58:41.712088 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:41.712051    5898 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 04:58:41.712485 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:41.712104    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl" podUID="58083aef-e47e-4ba5-84f0-a70d80973181" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 04:58:42.157541 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:42.157506    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:58:42.711480 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:42.711443    5898 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 04:58:42.711652 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:42.711504    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl" podUID="58083aef-e47e-4ba5-84f0-a70d80973181" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 04:58:43.715933 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:43.715883    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl"
Oct 30 04:58:43.716336 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:43.716179    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl"
Oct 30 04:58:43.716895 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:43.716877    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl"
Oct 30 04:58:43.847929 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:43.847868    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:58:45.882704 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:45.882673    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:58:47.545704 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:47.545666    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:58:48.603331 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:48.603299    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-324e5f78-rtx7p"]
Oct 30 04:58:48.606995 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:48.606974    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-324e5f78-rtx7p"
Oct 30 04:58:48.623940 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:48.623915    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-324e5f78-rtx7p"]
Oct 30 04:58:48.633171 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:48.633146    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:58:48.633302 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:48.633196    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\""
Oct 30 04:58:48.709822 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:48.709793    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4a8c37c-87a3-454d-a98d-db82229dc871-maas-api-xks-ca\") pod \"maas-api-revoke-sub-324e5f78-rtx7p\" (UID: \"f4a8c37c-87a3-454d-a98d-db82229dc871\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-324e5f78-rtx7p"
Oct 30 04:58:48.709997 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:48.709858    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jlhc\" (UniqueName: \"kubernetes.io/projected/f4a8c37c-87a3-454d-a98d-db82229dc871-kube-api-access-6jlhc\") pod \"maas-api-revoke-sub-324e5f78-rtx7p\" (UID: \"f4a8c37c-87a3-454d-a98d-db82229dc871\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-324e5f78-rtx7p"
Oct 30 04:58:48.709997 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:48.709984    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4a8c37c-87a3-454d-a98d-db82229dc871-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-324e5f78-rtx7p\" (UID: \"f4a8c37c-87a3-454d-a98d-db82229dc871\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-324e5f78-rtx7p"
Oct 30 04:58:48.811355 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:48.811315    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4a8c37c-87a3-454d-a98d-db82229dc871-maas-api-xks-ca\") pod \"maas-api-revoke-sub-324e5f78-rtx7p\" (UID: \"f4a8c37c-87a3-454d-a98d-db82229dc871\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-324e5f78-rtx7p"
Oct 30 04:58:48.811524 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:48.811384    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jlhc\" (UniqueName: \"kubernetes.io/projected/f4a8c37c-87a3-454d-a98d-db82229dc871-kube-api-access-6jlhc\") pod \"maas-api-revoke-sub-324e5f78-rtx7p\" (UID: \"f4a8c37c-87a3-454d-a98d-db82229dc871\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-324e5f78-rtx7p"
Oct 30 04:58:48.811524 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:48.811415    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4a8c37c-87a3-454d-a98d-db82229dc871-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-324e5f78-rtx7p\" (UID: \"f4a8c37c-87a3-454d-a98d-db82229dc871\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-324e5f78-rtx7p"
Oct 30 04:58:48.812111 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:48.812092    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4a8c37c-87a3-454d-a98d-db82229dc871-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-324e5f78-rtx7p\" (UID: \"f4a8c37c-87a3-454d-a98d-db82229dc871\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-324e5f78-rtx7p"
Oct 30 04:58:48.813707 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:48.813683    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4a8c37c-87a3-454d-a98d-db82229dc871-maas-api-xks-ca\") pod \"maas-api-revoke-sub-324e5f78-rtx7p\" (UID: \"f4a8c37c-87a3-454d-a98d-db82229dc871\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-324e5f78-rtx7p"
Oct 30 04:58:48.829848 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:48.829826    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jlhc\" (UniqueName: \"kubernetes.io/projected/f4a8c37c-87a3-454d-a98d-db82229dc871-kube-api-access-6jlhc\") pod \"maas-api-revoke-sub-324e5f78-rtx7p\" (UID: \"f4a8c37c-87a3-454d-a98d-db82229dc871\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-324e5f78-rtx7p"
Oct 30 04:58:48.920961 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:48.920870    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-324e5f78-rtx7p"
Oct 30 04:58:49.215596 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:49.215565    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-324e5f78-rtx7p"]
Oct 30 04:58:49.216944 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:58:49.216915    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4a8c37c_87a3_454d_a98d_db82229dc871.slice/crio-a0bf97dc2a48fb647e3053d2f844e3b9d0b9fdab17ed621288aeb1ff1b33af46 WatchSource:0}: Error finding container a0bf97dc2a48fb647e3053d2f844e3b9d0b9fdab17ed621288aeb1ff1b33af46: Status 404 returned error can't find the container with id a0bf97dc2a48fb647e3053d2f844e3b9d0b9fdab17ed621288aeb1ff1b33af46
Oct 30 04:58:49.609084 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:49.609052    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:58:49.728480 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:49.728444    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:58:49.786964 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:49.786871    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-324e5f78-rtx7p" event={"ID":"f4a8c37c-87a3-454d-a98d-db82229dc871","Type":"ContainerStarted","Data":"c7e908b9b884142eb55a420bd383d652fc30137d2ea9a907bbbcfb43f103ba2e"}
Oct 30 04:58:49.787120 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:49.786963    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-324e5f78-rtx7p" event={"ID":"f4a8c37c-87a3-454d-a98d-db82229dc871","Type":"ContainerStarted","Data":"a0bf97dc2a48fb647e3053d2f844e3b9d0b9fdab17ed621288aeb1ff1b33af46"}
Oct 30 04:58:49.834959 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:49.834870    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-324e5f78-rtx7p" podStartSLOduration=1.834853148 podStartE2EDuration="1.834853148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:58:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:58:49.832995539 +0000 UTC m=+859.056815461" watchObservedRunningTime="2026-10-30 04:58:49.834853148 +0000 UTC m=+859.058673054"
Oct 30 04:58:50.793752 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:50.793719    5898 generic.go:356] "Generic (PLEG): container finished" podID="f4a8c37c-87a3-454d-a98d-db82229dc871" containerID="c7e908b9b884142eb55a420bd383d652fc30137d2ea9a907bbbcfb43f103ba2e" exitCode=0
Oct 30 04:58:50.794179 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:50.793809    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-324e5f78-rtx7p" event={"ID":"f4a8c37c-87a3-454d-a98d-db82229dc871","Type":"ContainerDied","Data":"c7e908b9b884142eb55a420bd383d652fc30137d2ea9a907bbbcfb43f103ba2e"}
Oct 30 04:58:51.268032 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.267996    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr"]
Oct 30 04:58:51.271767 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.271747    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr"
Oct 30 04:58:51.287594 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.287574    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-397179-openshift-default-dockercfg-q4dsf\""
Oct 30 04:58:51.314387 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.314355    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr"]
Oct 30 04:58:51.336357 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.336325    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a5eec3-1d10-4860-bb38-33a092c84327-workload-socket\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr\" (UID: \"d1a5eec3-1d10-4860-bb38-33a092c84327\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr"
Oct 30 04:58:51.336500 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.336380    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1a5eec3-1d10-4860-bb38-33a092c84327-istio-token\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr\" (UID: \"d1a5eec3-1d10-4860-bb38-33a092c84327\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr"
Oct 30 04:58:51.336500 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.336458    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1a5eec3-1d10-4860-bb38-33a092c84327-istio-envoy\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr\" (UID: \"d1a5eec3-1d10-4860-bb38-33a092c84327\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr"
Oct 30 04:58:51.336500 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.336485    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1a5eec3-1d10-4860-bb38-33a092c84327-workload-certs\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr\" (UID: \"d1a5eec3-1d10-4860-bb38-33a092c84327\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr"
Oct 30 04:58:51.336650 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.336545    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a5eec3-1d10-4860-bb38-33a092c84327-credential-socket\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr\" (UID: \"d1a5eec3-1d10-4860-bb38-33a092c84327\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr"
Oct 30 04:58:51.336703 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.336687    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1a5eec3-1d10-4860-bb38-33a092c84327-istio-data\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr\" (UID: \"d1a5eec3-1d10-4860-bb38-33a092c84327\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr"
Oct 30 04:58:51.336756 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.336718    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1a5eec3-1d10-4860-bb38-33a092c84327-istiod-ca-cert\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr\" (UID: \"d1a5eec3-1d10-4860-bb38-33a092c84327\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr"
Oct 30 04:58:51.336817 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.336803    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1a5eec3-1d10-4860-bb38-33a092c84327-istio-podinfo\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr\" (UID: \"d1a5eec3-1d10-4860-bb38-33a092c84327\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr"
Oct 30 04:58:51.336877 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.336835    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbxsd\" (UniqueName: \"kubernetes.io/projected/d1a5eec3-1d10-4860-bb38-33a092c84327-kube-api-access-tbxsd\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr\" (UID: \"d1a5eec3-1d10-4860-bb38-33a092c84327\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr"
Oct 30 04:58:51.338832 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.338811    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:58:51.437716 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.437683    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a5eec3-1d10-4860-bb38-33a092c84327-credential-socket\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr\" (UID: \"d1a5eec3-1d10-4860-bb38-33a092c84327\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr"
Oct 30 04:58:51.437910 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.437747    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1a5eec3-1d10-4860-bb38-33a092c84327-istio-data\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr\" (UID: \"d1a5eec3-1d10-4860-bb38-33a092c84327\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr"
Oct 30 04:58:51.437910 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.437775    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1a5eec3-1d10-4860-bb38-33a092c84327-istiod-ca-cert\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr\" (UID: \"d1a5eec3-1d10-4860-bb38-33a092c84327\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr"
Oct 30 04:58:51.437910 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.437812    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1a5eec3-1d10-4860-bb38-33a092c84327-istio-podinfo\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr\" (UID: \"d1a5eec3-1d10-4860-bb38-33a092c84327\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr"
Oct 30 04:58:51.437910 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.437840    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbxsd\" (UniqueName: \"kubernetes.io/projected/d1a5eec3-1d10-4860-bb38-33a092c84327-kube-api-access-tbxsd\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr\" (UID: \"d1a5eec3-1d10-4860-bb38-33a092c84327\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr"
Oct 30 04:58:51.437910 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.437885    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a5eec3-1d10-4860-bb38-33a092c84327-workload-socket\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr\" (UID: \"d1a5eec3-1d10-4860-bb38-33a092c84327\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr"
Oct 30 04:58:51.438188 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.437950    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1a5eec3-1d10-4860-bb38-33a092c84327-istio-token\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr\" (UID: \"d1a5eec3-1d10-4860-bb38-33a092c84327\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr"
Oct 30 04:58:51.438188 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.437990    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1a5eec3-1d10-4860-bb38-33a092c84327-istio-envoy\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr\" (UID: \"d1a5eec3-1d10-4860-bb38-33a092c84327\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr"
Oct 30 04:58:51.438188 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.438016    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1a5eec3-1d10-4860-bb38-33a092c84327-workload-certs\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr\" (UID: \"d1a5eec3-1d10-4860-bb38-33a092c84327\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr"
Oct 30 04:58:51.438341 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.438236    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1a5eec3-1d10-4860-bb38-33a092c84327-istio-data\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr\" (UID: \"d1a5eec3-1d10-4860-bb38-33a092c84327\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr"
Oct 30 04:58:51.438395 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.438342    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a5eec3-1d10-4860-bb38-33a092c84327-workload-socket\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr\" (UID: \"d1a5eec3-1d10-4860-bb38-33a092c84327\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr"
Oct 30 04:58:51.438607 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.438586    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1a5eec3-1d10-4860-bb38-33a092c84327-istiod-ca-cert\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr\" (UID: \"d1a5eec3-1d10-4860-bb38-33a092c84327\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr"
Oct 30 04:58:51.438751 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.438718    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1a5eec3-1d10-4860-bb38-33a092c84327-workload-certs\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr\" (UID: \"d1a5eec3-1d10-4860-bb38-33a092c84327\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr"
Oct 30 04:58:51.438891 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.438871    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a5eec3-1d10-4860-bb38-33a092c84327-credential-socket\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr\" (UID: \"d1a5eec3-1d10-4860-bb38-33a092c84327\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr"
Oct 30 04:58:51.440237 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.440218    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1a5eec3-1d10-4860-bb38-33a092c84327-istio-envoy\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr\" (UID: \"d1a5eec3-1d10-4860-bb38-33a092c84327\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr"
Oct 30 04:58:51.440392 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.440375    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1a5eec3-1d10-4860-bb38-33a092c84327-istio-podinfo\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr\" (UID: \"d1a5eec3-1d10-4860-bb38-33a092c84327\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr"
Oct 30 04:58:51.459732 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.459705    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbxsd\" (UniqueName: \"kubernetes.io/projected/d1a5eec3-1d10-4860-bb38-33a092c84327-kube-api-access-tbxsd\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr\" (UID: \"d1a5eec3-1d10-4860-bb38-33a092c84327\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr"
Oct 30 04:58:51.460541 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.460521    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1a5eec3-1d10-4860-bb38-33a092c84327-istio-token\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr\" (UID: \"d1a5eec3-1d10-4860-bb38-33a092c84327\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr"
Oct 30 04:58:51.585415 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.585385    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr"
Oct 30 04:58:51.801019 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.800984    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr"]
Oct 30 04:58:51.804861 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.804843    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr"
Oct 30 04:58:51.834038 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.826467    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr"]
Oct 30 04:58:51.844987 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.841983    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c677544-d5bf-4024-8c23-15360f51b074-istio-token\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr\" (UID: \"3c677544-d5bf-4024-8c23-15360f51b074\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr"
Oct 30 04:58:51.844987 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.842030    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c677544-d5bf-4024-8c23-15360f51b074-workload-socket\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr\" (UID: \"3c677544-d5bf-4024-8c23-15360f51b074\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr"
Oct 30 04:58:51.844987 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.842065    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c677544-d5bf-4024-8c23-15360f51b074-istio-envoy\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr\" (UID: \"3c677544-d5bf-4024-8c23-15360f51b074\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr"
Oct 30 04:58:51.844987 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.842099    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c677544-d5bf-4024-8c23-15360f51b074-istiod-ca-cert\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr\" (UID: \"3c677544-d5bf-4024-8c23-15360f51b074\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr"
Oct 30 04:58:51.844987 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.842312    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c677544-d5bf-4024-8c23-15360f51b074-credential-socket\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr\" (UID: \"3c677544-d5bf-4024-8c23-15360f51b074\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr"
Oct 30 04:58:51.844987 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.842382    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c677544-d5bf-4024-8c23-15360f51b074-istio-data\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr\" (UID: \"3c677544-d5bf-4024-8c23-15360f51b074\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr"
Oct 30 04:58:51.844987 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.842409    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgrs7\" (UniqueName: \"kubernetes.io/projected/3c677544-d5bf-4024-8c23-15360f51b074-kube-api-access-jgrs7\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr\" (UID: \"3c677544-d5bf-4024-8c23-15360f51b074\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr"
Oct 30 04:58:51.844987 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.842521    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c677544-d5bf-4024-8c23-15360f51b074-istio-podinfo\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr\" (UID: \"3c677544-d5bf-4024-8c23-15360f51b074\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr"
Oct 30 04:58:51.844987 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.842566    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c677544-d5bf-4024-8c23-15360f51b074-workload-certs\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr\" (UID: \"3c677544-d5bf-4024-8c23-15360f51b074\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr"
Oct 30 04:58:51.911056 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:58:51.911016    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1a5eec3_1d10_4860_bb38_33a092c84327.slice/crio-6b3973695c42b741027290d7223d2aab58621b7fed657b7f0a22848e6243f135 WatchSource:0}: Error finding container 6b3973695c42b741027290d7223d2aab58621b7fed657b7f0a22848e6243f135: Status 404 returned error can't find the container with id 6b3973695c42b741027290d7223d2aab58621b7fed657b7f0a22848e6243f135
Oct 30 04:58:51.911308 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.911279    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-cff4ac-openshift-default-dockercfg-xcpvh\""
Oct 30 04:58:51.913124 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.913090    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 04:58:51.913244 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.913154    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 04:58:51.913244 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.913189    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 04:58:51.914518 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.914492    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr"]
Oct 30 04:58:51.943995 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.943973    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c677544-d5bf-4024-8c23-15360f51b074-workload-certs\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr\" (UID: \"3c677544-d5bf-4024-8c23-15360f51b074\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr"
Oct 30 04:58:51.944109 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.944021    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c677544-d5bf-4024-8c23-15360f51b074-istio-token\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr\" (UID: \"3c677544-d5bf-4024-8c23-15360f51b074\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr"
Oct 30 04:58:51.944109 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.944065    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c677544-d5bf-4024-8c23-15360f51b074-workload-socket\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr\" (UID: \"3c677544-d5bf-4024-8c23-15360f51b074\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr"
Oct 30 04:58:51.944109 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.944091    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c677544-d5bf-4024-8c23-15360f51b074-istio-envoy\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr\" (UID: \"3c677544-d5bf-4024-8c23-15360f51b074\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr"
Oct 30 04:58:51.944256 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.944116    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c677544-d5bf-4024-8c23-15360f51b074-istiod-ca-cert\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr\" (UID: \"3c677544-d5bf-4024-8c23-15360f51b074\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr"
Oct 30 04:58:51.944256 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.944161    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c677544-d5bf-4024-8c23-15360f51b074-credential-socket\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr\" (UID: \"3c677544-d5bf-4024-8c23-15360f51b074\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr"
Oct 30 04:58:51.944256 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.944246    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c677544-d5bf-4024-8c23-15360f51b074-istio-data\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr\" (UID: \"3c677544-d5bf-4024-8c23-15360f51b074\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr"
Oct 30 04:58:51.944413 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.944276    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgrs7\" (UniqueName: \"kubernetes.io/projected/3c677544-d5bf-4024-8c23-15360f51b074-kube-api-access-jgrs7\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr\" (UID: \"3c677544-d5bf-4024-8c23-15360f51b074\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr"
Oct 30 04:58:51.944413 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.944348    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c677544-d5bf-4024-8c23-15360f51b074-istio-podinfo\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr\" (UID: \"3c677544-d5bf-4024-8c23-15360f51b074\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr"
Oct 30 04:58:51.944556 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.944536    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c677544-d5bf-4024-8c23-15360f51b074-workload-certs\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr\" (UID: \"3c677544-d5bf-4024-8c23-15360f51b074\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr"
Oct 30 04:58:51.944625 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.944601    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c677544-d5bf-4024-8c23-15360f51b074-workload-socket\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr\" (UID: \"3c677544-d5bf-4024-8c23-15360f51b074\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr"
Oct 30 04:58:51.944681 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.944645    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c677544-d5bf-4024-8c23-15360f51b074-istio-data\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr\" (UID: \"3c677544-d5bf-4024-8c23-15360f51b074\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr"
Oct 30 04:58:51.944991 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.944968    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c677544-d5bf-4024-8c23-15360f51b074-credential-socket\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr\" (UID: \"3c677544-d5bf-4024-8c23-15360f51b074\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr"
Oct 30 04:58:51.945337 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.945319    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c677544-d5bf-4024-8c23-15360f51b074-istiod-ca-cert\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr\" (UID: \"3c677544-d5bf-4024-8c23-15360f51b074\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr"
Oct 30 04:58:51.946728 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.946702    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c677544-d5bf-4024-8c23-15360f51b074-istio-envoy\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr\" (UID: \"3c677544-d5bf-4024-8c23-15360f51b074\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr"
Oct 30 04:58:51.947108 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.947092    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c677544-d5bf-4024-8c23-15360f51b074-istio-podinfo\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr\" (UID: \"3c677544-d5bf-4024-8c23-15360f51b074\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr"
Oct 30 04:58:51.953447 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.953430    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-324e5f78-rtx7p"
Oct 30 04:58:51.962484 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.962461    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c677544-d5bf-4024-8c23-15360f51b074-istio-token\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr\" (UID: \"3c677544-d5bf-4024-8c23-15360f51b074\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr"
Oct 30 04:58:51.962754 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:51.962729    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgrs7\" (UniqueName: \"kubernetes.io/projected/3c677544-d5bf-4024-8c23-15360f51b074-kube-api-access-jgrs7\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr\" (UID: \"3c677544-d5bf-4024-8c23-15360f51b074\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr"
Oct 30 04:58:52.044847 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:52.044816    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4a8c37c-87a3-454d-a98d-db82229dc871-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4a8c37c-87a3-454d-a98d-db82229dc871-maas-api-openshift-ca\") pod \"f4a8c37c-87a3-454d-a98d-db82229dc871\" (UID: \"f4a8c37c-87a3-454d-a98d-db82229dc871\") "
Oct 30 04:58:52.045051 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:52.044883    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4a8c37c-87a3-454d-a98d-db82229dc871-kube-api-access-6jlhc\" (UniqueName: \"kubernetes.io/projected/f4a8c37c-87a3-454d-a98d-db82229dc871-kube-api-access-6jlhc\") pod \"f4a8c37c-87a3-454d-a98d-db82229dc871\" (UID: \"f4a8c37c-87a3-454d-a98d-db82229dc871\") "
Oct 30 04:58:52.045051 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:52.044966    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4a8c37c-87a3-454d-a98d-db82229dc871-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4a8c37c-87a3-454d-a98d-db82229dc871-maas-api-xks-ca\") pod \"f4a8c37c-87a3-454d-a98d-db82229dc871\" (UID: \"f4a8c37c-87a3-454d-a98d-db82229dc871\") "
Oct 30 04:58:52.045235 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:52.045206    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4a8c37c-87a3-454d-a98d-db82229dc871-maas-api-openshift-ca" pod "f4a8c37c-87a3-454d-a98d-db82229dc871" (UID: "f4a8c37c-87a3-454d-a98d-db82229dc871"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:58:52.046941 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:52.046887    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4a8c37c-87a3-454d-a98d-db82229dc871-maas-api-xks-ca" pod "f4a8c37c-87a3-454d-a98d-db82229dc871" (UID: "f4a8c37c-87a3-454d-a98d-db82229dc871"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:58:52.047047 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:52.047024    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4a8c37c-87a3-454d-a98d-db82229dc871-kube-api-access-6jlhc" pod "f4a8c37c-87a3-454d-a98d-db82229dc871" (UID: "f4a8c37c-87a3-454d-a98d-db82229dc871"). InnerVolumeSpecName "kube-api-access-6jlhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:58:52.138189 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:52.138116    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr"
Oct 30 04:58:52.146865 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:52.146836    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4a8c37c-87a3-454d-a98d-db82229dc871-maas-api-xks-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 04:58:52.146987 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:52.146870    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4a8c37c-87a3-454d-a98d-db82229dc871-maas-api-openshift-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 04:58:52.146987 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:52.146884    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jlhc\" (UniqueName: \"kubernetes.io/projected/f4a8c37c-87a3-454d-a98d-db82229dc871-kube-api-access-6jlhc\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 04:58:52.454197 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:52.454164    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr"]
Oct 30 04:58:52.458854 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:58:52.458820    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c677544_d5bf_4024_8c23_15360f51b074.slice/crio-4b8bbe7690033842a9dafc3d29a66f25ad73a11dd27000d982a2ecfb85a7f69c WatchSource:0}: Error finding container 4b8bbe7690033842a9dafc3d29a66f25ad73a11dd27000d982a2ecfb85a7f69c: Status 404 returned error can't find the container with id 4b8bbe7690033842a9dafc3d29a66f25ad73a11dd27000d982a2ecfb85a7f69c
Oct 30 04:58:52.461665 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:52.461634    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 04:58:52.461762 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:52.461713    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 04:58:52.461831 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:52.461757    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 04:58:52.809806 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:52.809703    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr" event={"ID":"d1a5eec3-1d10-4860-bb38-33a092c84327","Type":"ContainerStarted","Data":"9ecf01636d6da6d91cf8d450bf43203a6b86eb1e842d3c355c4e4e7a9443c123"}
Oct 30 04:58:52.809806 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:52.809777    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr" event={"ID":"d1a5eec3-1d10-4860-bb38-33a092c84327","Type":"ContainerStarted","Data":"6b3973695c42b741027290d7223d2aab58621b7fed657b7f0a22848e6243f135"}
Oct 30 04:58:52.811277 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:52.811256    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-324e5f78-rtx7p"
Oct 30 04:58:52.811387 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:52.811275    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-324e5f78-rtx7p" event={"ID":"f4a8c37c-87a3-454d-a98d-db82229dc871","Type":"ContainerDied","Data":"a0bf97dc2a48fb647e3053d2f844e3b9d0b9fdab17ed621288aeb1ff1b33af46"}
Oct 30 04:58:52.811387 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:52.811321    5898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0bf97dc2a48fb647e3053d2f844e3b9d0b9fdab17ed621288aeb1ff1b33af46"
Oct 30 04:58:52.813109 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:52.813082    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr" event={"ID":"3c677544-d5bf-4024-8c23-15360f51b074","Type":"ContainerStarted","Data":"666ca71863b326852fe3e3e5c461b4b1f1c6e025038dd578b39e6b8c726369ef"}
Oct 30 04:58:52.813210 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:52.813139    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr" event={"ID":"3c677544-d5bf-4024-8c23-15360f51b074","Type":"ContainerStarted","Data":"4b8bbe7690033842a9dafc3d29a66f25ad73a11dd27000d982a2ecfb85a7f69c"}
Oct 30 04:58:52.879937 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:52.873946    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr" podStartSLOduration=1.873928671 podStartE2EDuration="1.873928671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:58:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:58:52.860083252 +0000 UTC m=+862.083903176" watchObservedRunningTime="2026-10-30 04:58:52.873928671 +0000 UTC m=+862.097748574"
Oct 30 04:58:52.915553 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:52.915502    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr" podStartSLOduration=1.915488429 podStartE2EDuration="1.915488429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:58:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:58:52.914466118 +0000 UTC m=+862.138286038" watchObservedRunningTime="2026-10-30 04:58:52.915488429 +0000 UTC m=+862.139308329"
Oct 30 04:58:53.138852 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:53.138814    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr"
Oct 30 04:58:53.140522 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:53.140500    5898 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 04:58:53.140581 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:53.140546    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr" podUID="3c677544-d5bf-4024-8c23-15360f51b074" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 04:58:53.281926 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:53.278865    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:58:53.589925 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:53.585961    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr"
Oct 30 04:58:53.615260 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:53.615208    5898 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 04:58:53.615569 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:53.615523    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr" podUID="d1a5eec3-1d10-4860-bb38-33a092c84327" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 04:58:54.139224 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:54.139192    5898 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 04:58:54.139619 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:54.139252    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr" podUID="3c677544-d5bf-4024-8c23-15360f51b074" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 04:58:54.587927 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:54.587200    5898 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 04:58:54.587927 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:54.587259    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr" podUID="d1a5eec3-1d10-4860-bb38-33a092c84327" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 04:58:55.139161 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:55.139117    5898 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 04:58:55.139348 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:55.139175    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr" podUID="3c677544-d5bf-4024-8c23-15360f51b074" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 04:58:55.234965 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:55.234923    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:58:55.587310 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:55.587272    5898 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 04:58:55.587569 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:55.587542    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr" podUID="d1a5eec3-1d10-4860-bb38-33a092c84327" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 04:58:56.139136 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:56.139097    5898 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 04:58:56.139293 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:56.139154    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr" podUID="3c677544-d5bf-4024-8c23-15360f51b074" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 04:58:56.596968 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:56.596932    5898 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 04:58:56.597528 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:56.597499    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr" podUID="d1a5eec3-1d10-4860-bb38-33a092c84327" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 04:58:56.847773 ip-10-0-135-98 kubenswrapper[5898]: E1030 04:58:56.847688    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 04:58:57.139079 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:57.138985    5898 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 04:58:57.139079 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:57.139050    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr" podUID="3c677544-d5bf-4024-8c23-15360f51b074" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 04:58:57.215988 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:57.215950    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 04:58:57.417984 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:57.413973    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:58:57.546009 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:57.545970    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-521287b6-tgxzz"]
Oct 30 04:58:57.546406 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:57.546393    5898 cpu_manager.go:414] "containerMap: removing container" podUID="f4a8c37c-87a3-454d-a98d-db82229dc871" containerName="revoke-keys"
Oct 30 04:58:57.546453 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:57.546407    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4a8c37c-87a3-454d-a98d-db82229dc871" containerName="revoke-keys"
Oct 30 04:58:57.546487 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:57.546480    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4a8c37c-87a3-454d-a98d-db82229dc871" containerName="revoke-keys"
Oct 30 04:58:57.549959 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:57.549937    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-521287b6-tgxzz"
Oct 30 04:58:57.564397 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:57.564377    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:58:57.564535 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:57.564481    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\""
Oct 30 04:58:57.565840 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:57.565817    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-521287b6-tgxzz"]
Oct 30 04:58:57.605314 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:57.605278    5898 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 04:58:57.639148 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:57.632987    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr" podUID="d1a5eec3-1d10-4860-bb38-33a092c84327" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 04:58:57.644146 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:57.644121    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/336ce57a-6ab1-41bd-9c74-6ba31bd6e72b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-521287b6-tgxzz\" (UID: \"336ce57a-6ab1-41bd-9c74-6ba31bd6e72b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-521287b6-tgxzz"
Oct 30 04:58:57.644367 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:57.644353    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddx82\" (UniqueName: \"kubernetes.io/projected/336ce57a-6ab1-41bd-9c74-6ba31bd6e72b-kube-api-access-ddx82\") pod \"maas-api-revoke-sub-521287b6-tgxzz\" (UID: \"336ce57a-6ab1-41bd-9c74-6ba31bd6e72b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-521287b6-tgxzz"
Oct 30 04:58:57.644532 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:57.644518    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/336ce57a-6ab1-41bd-9c74-6ba31bd6e72b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-521287b6-tgxzz\" (UID: \"336ce57a-6ab1-41bd-9c74-6ba31bd6e72b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-521287b6-tgxzz"
Oct 30 04:58:57.745740 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:57.745651    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/336ce57a-6ab1-41bd-9c74-6ba31bd6e72b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-521287b6-tgxzz\" (UID: \"336ce57a-6ab1-41bd-9c74-6ba31bd6e72b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-521287b6-tgxzz"
Oct 30 04:58:57.746101 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:57.746076    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/336ce57a-6ab1-41bd-9c74-6ba31bd6e72b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-521287b6-tgxzz\" (UID: \"336ce57a-6ab1-41bd-9c74-6ba31bd6e72b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-521287b6-tgxzz"
Oct 30 04:58:57.746673 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:57.746647    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/336ce57a-6ab1-41bd-9c74-6ba31bd6e72b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-521287b6-tgxzz\" (UID: \"336ce57a-6ab1-41bd-9c74-6ba31bd6e72b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-521287b6-tgxzz"
Oct 30 04:58:57.746805 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:57.746787    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddx82\" (UniqueName: \"kubernetes.io/projected/336ce57a-6ab1-41bd-9c74-6ba31bd6e72b-kube-api-access-ddx82\") pod \"maas-api-revoke-sub-521287b6-tgxzz\" (UID: \"336ce57a-6ab1-41bd-9c74-6ba31bd6e72b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-521287b6-tgxzz"
Oct 30 04:58:57.748783 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:57.748757    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/336ce57a-6ab1-41bd-9c74-6ba31bd6e72b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-521287b6-tgxzz\" (UID: \"336ce57a-6ab1-41bd-9c74-6ba31bd6e72b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-521287b6-tgxzz"
Oct 30 04:58:57.767342 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:57.767309    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddx82\" (UniqueName: \"kubernetes.io/projected/336ce57a-6ab1-41bd-9c74-6ba31bd6e72b-kube-api-access-ddx82\") pod \"maas-api-revoke-sub-521287b6-tgxzz\" (UID: \"336ce57a-6ab1-41bd-9c74-6ba31bd6e72b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-521287b6-tgxzz"
Oct 30 04:58:57.863098 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:57.863061    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-521287b6-tgxzz"
Oct 30 04:58:58.139537 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:58.139078    5898 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 04:58:58.139537 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:58.139134    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr" podUID="3c677544-d5bf-4024-8c23-15360f51b074" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 04:58:58.258977 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:58.235195    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-521287b6-tgxzz"]
Oct 30 04:58:58.590306 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:58.590273    5898 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 04:58:58.590621 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:58.590572    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr" podUID="d1a5eec3-1d10-4860-bb38-33a092c84327" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 04:58:58.863985 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:58.863947    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-521287b6-tgxzz" event={"ID":"336ce57a-6ab1-41bd-9c74-6ba31bd6e72b","Type":"ContainerStarted","Data":"99c74e91f6bc755caa3754c5fd7dde2a8b4b7319031d1349fcb3c45d2de7c116"}
Oct 30 04:58:58.864461 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:58.864030    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-521287b6-tgxzz" event={"ID":"336ce57a-6ab1-41bd-9c74-6ba31bd6e72b","Type":"ContainerStarted","Data":"bbdf2dea9501cc535063820c1ba3e7cc3b90e2598f63fcd7e166635ff70c6eee"}
Oct 30 04:58:59.887923 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:59.887480    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:58:59.895922 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:59.893427    5898 generic.go:356] "Generic (PLEG): container finished" podID="336ce57a-6ab1-41bd-9c74-6ba31bd6e72b" containerID="99c74e91f6bc755caa3754c5fd7dde2a8b4b7319031d1349fcb3c45d2de7c116" exitCode=0
Oct 30 04:58:59.895922 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:58:59.893627    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-521287b6-tgxzz" event={"ID":"336ce57a-6ab1-41bd-9c74-6ba31bd6e72b","Type":"ContainerDied","Data":"99c74e91f6bc755caa3754c5fd7dde2a8b4b7319031d1349fcb3c45d2de7c116"}
Oct 30 04:59:00.141302 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:00.141213    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr"
Oct 30 04:59:00.142813 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:00.142792    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr"
Oct 30 04:59:00.150182 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:00.150131    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr"
Oct 30 04:59:00.216377 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:00.216341    5898 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 04:59:00.216669 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:00.216641    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr" podUID="3c677544-d5bf-4024-8c23-15360f51b074" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 04:59:00.565753 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:00.565674    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr"
Oct 30 04:59:00.567266 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:00.567230    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr"
Oct 30 04:59:00.569654 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:00.569611    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr"
Oct 30 04:59:01.096930 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:01.096274    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-521287b6-tgxzz"
Oct 30 04:59:01.244772 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:01.244691    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/336ce57a-6ab1-41bd-9c74-6ba31bd6e72b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/336ce57a-6ab1-41bd-9c74-6ba31bd6e72b-maas-api-xks-ca\") pod \"336ce57a-6ab1-41bd-9c74-6ba31bd6e72b\" (UID: \"336ce57a-6ab1-41bd-9c74-6ba31bd6e72b\") "
Oct 30 04:59:01.244772 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:01.244765    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/336ce57a-6ab1-41bd-9c74-6ba31bd6e72b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/336ce57a-6ab1-41bd-9c74-6ba31bd6e72b-maas-api-openshift-ca\") pod \"336ce57a-6ab1-41bd-9c74-6ba31bd6e72b\" (UID: \"336ce57a-6ab1-41bd-9c74-6ba31bd6e72b\") "
Oct 30 04:59:01.244986 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:01.244882    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/336ce57a-6ab1-41bd-9c74-6ba31bd6e72b-kube-api-access-ddx82\" (UniqueName: \"kubernetes.io/projected/336ce57a-6ab1-41bd-9c74-6ba31bd6e72b-kube-api-access-ddx82\") pod \"336ce57a-6ab1-41bd-9c74-6ba31bd6e72b\" (UID: \"336ce57a-6ab1-41bd-9c74-6ba31bd6e72b\") "
Oct 30 04:59:01.245107 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:01.245076    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/336ce57a-6ab1-41bd-9c74-6ba31bd6e72b-maas-api-openshift-ca" pod "336ce57a-6ab1-41bd-9c74-6ba31bd6e72b" (UID: "336ce57a-6ab1-41bd-9c74-6ba31bd6e72b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:59:01.245268 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:01.245253    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/336ce57a-6ab1-41bd-9c74-6ba31bd6e72b-maas-api-openshift-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 04:59:01.246739 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:01.246719    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/336ce57a-6ab1-41bd-9c74-6ba31bd6e72b-maas-api-xks-ca" pod "336ce57a-6ab1-41bd-9c74-6ba31bd6e72b" (UID: "336ce57a-6ab1-41bd-9c74-6ba31bd6e72b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:59:01.246817 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:01.246796    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/336ce57a-6ab1-41bd-9c74-6ba31bd6e72b-kube-api-access-ddx82" pod "336ce57a-6ab1-41bd-9c74-6ba31bd6e72b" (UID: "336ce57a-6ab1-41bd-9c74-6ba31bd6e72b"). InnerVolumeSpecName "kube-api-access-ddx82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:59:01.345823 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:01.345784    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/336ce57a-6ab1-41bd-9c74-6ba31bd6e72b-maas-api-xks-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 04:59:01.345823 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:01.345817    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddx82\" (UniqueName: \"kubernetes.io/projected/336ce57a-6ab1-41bd-9c74-6ba31bd6e72b-kube-api-access-ddx82\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 04:59:01.909259 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:01.909230    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-521287b6-tgxzz"
Oct 30 04:59:01.909259 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:01.909244    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-521287b6-tgxzz" event={"ID":"336ce57a-6ab1-41bd-9c74-6ba31bd6e72b","Type":"ContainerDied","Data":"bbdf2dea9501cc535063820c1ba3e7cc3b90e2598f63fcd7e166635ff70c6eee"}
Oct 30 04:59:01.909468 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:01.909292    5898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bbdf2dea9501cc535063820c1ba3e7cc3b90e2598f63fcd7e166635ff70c6eee"
Oct 30 04:59:01.964695 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:01.964662    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:59:03.666129 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:03.666091    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:59:05.275246 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:05.275212    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:59:06.991554 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:06.991520    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw"]
Oct 30 04:59:06.992158 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:06.992138    5898 cpu_manager.go:414] "containerMap: removing container" podUID="336ce57a-6ab1-41bd-9c74-6ba31bd6e72b" containerName="revoke-keys"
Oct 30 04:59:06.992239 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:06.992160    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="336ce57a-6ab1-41bd-9c74-6ba31bd6e72b" containerName="revoke-keys"
Oct 30 04:59:06.992295 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:06.992282    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="336ce57a-6ab1-41bd-9c74-6ba31bd6e72b" containerName="revoke-keys"
Oct 30 04:59:06.999649 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:06.999626    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw"
Oct 30 04:59:07.018356 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.018333    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw"]
Oct 30 04:59:07.027259 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.027233    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-0b21ba-openshift-default-dockercfg-5n9kc\""
Oct 30 04:59:07.078111 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.078083    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:59:07.106664 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.106626    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cdfbc2d-3a92-4782-89cb-886f47416486-credential-socket\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw\" (UID: \"8cdfbc2d-3a92-4782-89cb-886f47416486\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw"
Oct 30 04:59:07.106819 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.106692    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cdfbc2d-3a92-4782-89cb-886f47416486-istio-token\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw\" (UID: \"8cdfbc2d-3a92-4782-89cb-886f47416486\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw"
Oct 30 04:59:07.106819 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.106796    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cdfbc2d-3a92-4782-89cb-886f47416486-istio-envoy\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw\" (UID: \"8cdfbc2d-3a92-4782-89cb-886f47416486\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw"
Oct 30 04:59:07.106988 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.106836    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cdfbc2d-3a92-4782-89cb-886f47416486-istio-podinfo\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw\" (UID: \"8cdfbc2d-3a92-4782-89cb-886f47416486\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw"
Oct 30 04:59:07.106988 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.106937    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cdfbc2d-3a92-4782-89cb-886f47416486-istio-data\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw\" (UID: \"8cdfbc2d-3a92-4782-89cb-886f47416486\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw"
Oct 30 04:59:07.106988 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.106969    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cdfbc2d-3a92-4782-89cb-886f47416486-istiod-ca-cert\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw\" (UID: \"8cdfbc2d-3a92-4782-89cb-886f47416486\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw"
Oct 30 04:59:07.107108 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.107005    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cdfbc2d-3a92-4782-89cb-886f47416486-workload-socket\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw\" (UID: \"8cdfbc2d-3a92-4782-89cb-886f47416486\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw"
Oct 30 04:59:07.107147 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.107100    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mfc2\" (UniqueName: \"kubernetes.io/projected/8cdfbc2d-3a92-4782-89cb-886f47416486-kube-api-access-7mfc2\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw\" (UID: \"8cdfbc2d-3a92-4782-89cb-886f47416486\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw"
Oct 30 04:59:07.107147 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.107139    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cdfbc2d-3a92-4782-89cb-886f47416486-workload-certs\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw\" (UID: \"8cdfbc2d-3a92-4782-89cb-886f47416486\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw"
Oct 30 04:59:07.126910 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.126873    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9"]
Oct 30 04:59:07.130782 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.130763    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9"
Oct 30 04:59:07.147239 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.147214    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-d55037-openshift-default-dockercfg-ppf9l\""
Oct 30 04:59:07.151115 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.151086    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9"]
Oct 30 04:59:07.208661 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.208632    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cdfbc2d-3a92-4782-89cb-886f47416486-istio-token\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw\" (UID: \"8cdfbc2d-3a92-4782-89cb-886f47416486\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw"
Oct 30 04:59:07.208827 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.208671    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfe8c234-c443-44a6-8420-9566768d6d02-istio-podinfo\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9\" (UID: \"bfe8c234-c443-44a6-8420-9566768d6d02\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9"
Oct 30 04:59:07.208827 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.208701    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cdfbc2d-3a92-4782-89cb-886f47416486-istio-envoy\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw\" (UID: \"8cdfbc2d-3a92-4782-89cb-886f47416486\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw"
Oct 30 04:59:07.208827 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.208765    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cdfbc2d-3a92-4782-89cb-886f47416486-istio-podinfo\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw\" (UID: \"8cdfbc2d-3a92-4782-89cb-886f47416486\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw"
Oct 30 04:59:07.208827 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.208810    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe8c234-c443-44a6-8420-9566768d6d02-credential-socket\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9\" (UID: \"bfe8c234-c443-44a6-8420-9566768d6d02\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9"
Oct 30 04:59:07.209031 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.208836    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cdfbc2d-3a92-4782-89cb-886f47416486-istio-data\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw\" (UID: \"8cdfbc2d-3a92-4782-89cb-886f47416486\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw"
Oct 30 04:59:07.209031 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.208858    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cdfbc2d-3a92-4782-89cb-886f47416486-istiod-ca-cert\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw\" (UID: \"8cdfbc2d-3a92-4782-89cb-886f47416486\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw"
Oct 30 04:59:07.209031 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.208883    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cdfbc2d-3a92-4782-89cb-886f47416486-workload-socket\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw\" (UID: \"8cdfbc2d-3a92-4782-89cb-886f47416486\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw"
Oct 30 04:59:07.209031 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.209005    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe8c234-c443-44a6-8420-9566768d6d02-workload-socket\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9\" (UID: \"bfe8c234-c443-44a6-8420-9566768d6d02\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9"
Oct 30 04:59:07.209241 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.209034    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfe8c234-c443-44a6-8420-9566768d6d02-istiod-ca-cert\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9\" (UID: \"bfe8c234-c443-44a6-8420-9566768d6d02\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9"
Oct 30 04:59:07.209241 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.209075    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mfc2\" (UniqueName: \"kubernetes.io/projected/8cdfbc2d-3a92-4782-89cb-886f47416486-kube-api-access-7mfc2\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw\" (UID: \"8cdfbc2d-3a92-4782-89cb-886f47416486\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw"
Oct 30 04:59:07.209365 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.209258    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cdfbc2d-3a92-4782-89cb-886f47416486-workload-certs\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw\" (UID: \"8cdfbc2d-3a92-4782-89cb-886f47416486\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw"
Oct 30 04:59:07.209365 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.209312    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfe8c234-c443-44a6-8420-9566768d6d02-istio-token\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9\" (UID: \"bfe8c234-c443-44a6-8420-9566768d6d02\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9"
Oct 30 04:59:07.209365 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.209330    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cdfbc2d-3a92-4782-89cb-886f47416486-istio-data\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw\" (UID: \"8cdfbc2d-3a92-4782-89cb-886f47416486\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw"
Oct 30 04:59:07.209365 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.209340    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfe8c234-c443-44a6-8420-9566768d6d02-istio-envoy\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9\" (UID: \"bfe8c234-c443-44a6-8420-9566768d6d02\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9"
Oct 30 04:59:07.209589 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.209406    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cdfbc2d-3a92-4782-89cb-886f47416486-workload-socket\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw\" (UID: \"8cdfbc2d-3a92-4782-89cb-886f47416486\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw"
Oct 30 04:59:07.209589 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.209465    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cdfbc2d-3a92-4782-89cb-886f47416486-credential-socket\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw\" (UID: \"8cdfbc2d-3a92-4782-89cb-886f47416486\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw"
Oct 30 04:59:07.209589 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.209520    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk8kc\" (UniqueName: \"kubernetes.io/projected/bfe8c234-c443-44a6-8420-9566768d6d02-kube-api-access-pk8kc\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9\" (UID: \"bfe8c234-c443-44a6-8420-9566768d6d02\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9"
Oct 30 04:59:07.209589 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.209576    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfe8c234-c443-44a6-8420-9566768d6d02-workload-certs\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9\" (UID: \"bfe8c234-c443-44a6-8420-9566768d6d02\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9"
Oct 30 04:59:07.209589 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.209580    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cdfbc2d-3a92-4782-89cb-886f47416486-workload-certs\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw\" (UID: \"8cdfbc2d-3a92-4782-89cb-886f47416486\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw"
Oct 30 04:59:07.209813 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.209670    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfe8c234-c443-44a6-8420-9566768d6d02-istio-data\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9\" (UID: \"bfe8c234-c443-44a6-8420-9566768d6d02\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9"
Oct 30 04:59:07.209813 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.209706    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cdfbc2d-3a92-4782-89cb-886f47416486-istiod-ca-cert\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw\" (UID: \"8cdfbc2d-3a92-4782-89cb-886f47416486\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw"
Oct 30 04:59:07.209813 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.209755    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cdfbc2d-3a92-4782-89cb-886f47416486-credential-socket\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw\" (UID: \"8cdfbc2d-3a92-4782-89cb-886f47416486\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw"
Oct 30 04:59:07.211712 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.211689    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cdfbc2d-3a92-4782-89cb-886f47416486-istio-envoy\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw\" (UID: \"8cdfbc2d-3a92-4782-89cb-886f47416486\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw"
Oct 30 04:59:07.212051 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.212029    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cdfbc2d-3a92-4782-89cb-886f47416486-istio-podinfo\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw\" (UID: \"8cdfbc2d-3a92-4782-89cb-886f47416486\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw"
Oct 30 04:59:07.228197 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.228171    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cdfbc2d-3a92-4782-89cb-886f47416486-istio-token\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw\" (UID: \"8cdfbc2d-3a92-4782-89cb-886f47416486\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw"
Oct 30 04:59:07.228301 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.228275    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mfc2\" (UniqueName: \"kubernetes.io/projected/8cdfbc2d-3a92-4782-89cb-886f47416486-kube-api-access-7mfc2\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw\" (UID: \"8cdfbc2d-3a92-4782-89cb-886f47416486\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw"
Oct 30 04:59:07.310738 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.310664    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfe8c234-c443-44a6-8420-9566768d6d02-workload-certs\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9\" (UID: \"bfe8c234-c443-44a6-8420-9566768d6d02\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9"
Oct 30 04:59:07.310738 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.310697    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfe8c234-c443-44a6-8420-9566768d6d02-istio-data\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9\" (UID: \"bfe8c234-c443-44a6-8420-9566768d6d02\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9"
Oct 30 04:59:07.310738 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.310737    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfe8c234-c443-44a6-8420-9566768d6d02-istio-podinfo\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9\" (UID: \"bfe8c234-c443-44a6-8420-9566768d6d02\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9"
Oct 30 04:59:07.311052 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.310768    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe8c234-c443-44a6-8420-9566768d6d02-credential-socket\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9\" (UID: \"bfe8c234-c443-44a6-8420-9566768d6d02\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9"
Oct 30 04:59:07.311052 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.310801    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe8c234-c443-44a6-8420-9566768d6d02-workload-socket\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9\" (UID: \"bfe8c234-c443-44a6-8420-9566768d6d02\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9"
Oct 30 04:59:07.311052 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.310818    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfe8c234-c443-44a6-8420-9566768d6d02-istiod-ca-cert\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9\" (UID: \"bfe8c234-c443-44a6-8420-9566768d6d02\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9"
Oct 30 04:59:07.311052 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.310879    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfe8c234-c443-44a6-8420-9566768d6d02-istio-token\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9\" (UID: \"bfe8c234-c443-44a6-8420-9566768d6d02\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9"
Oct 30 04:59:07.311052 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.310911    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfe8c234-c443-44a6-8420-9566768d6d02-istio-envoy\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9\" (UID: \"bfe8c234-c443-44a6-8420-9566768d6d02\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9"
Oct 30 04:59:07.311052 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.310965    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk8kc\" (UniqueName: \"kubernetes.io/projected/bfe8c234-c443-44a6-8420-9566768d6d02-kube-api-access-pk8kc\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9\" (UID: \"bfe8c234-c443-44a6-8420-9566768d6d02\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9"
Oct 30 04:59:07.311373 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.311136    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfe8c234-c443-44a6-8420-9566768d6d02-workload-certs\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9\" (UID: \"bfe8c234-c443-44a6-8420-9566768d6d02\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9"
Oct 30 04:59:07.311373 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.311281    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe8c234-c443-44a6-8420-9566768d6d02-credential-socket\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9\" (UID: \"bfe8c234-c443-44a6-8420-9566768d6d02\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9"
Oct 30 04:59:07.311373 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.311314    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfe8c234-c443-44a6-8420-9566768d6d02-istio-data\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9\" (UID: \"bfe8c234-c443-44a6-8420-9566768d6d02\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9"
Oct 30 04:59:07.311585 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.311564    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe8c234-c443-44a6-8420-9566768d6d02-workload-socket\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9\" (UID: \"bfe8c234-c443-44a6-8420-9566768d6d02\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9"
Oct 30 04:59:07.311663 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.311584    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfe8c234-c443-44a6-8420-9566768d6d02-istiod-ca-cert\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9\" (UID: \"bfe8c234-c443-44a6-8420-9566768d6d02\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9"
Oct 30 04:59:07.312501 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.312480    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw"
Oct 30 04:59:07.313101 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.313085    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfe8c234-c443-44a6-8420-9566768d6d02-istio-envoy\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9\" (UID: \"bfe8c234-c443-44a6-8420-9566768d6d02\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9"
Oct 30 04:59:07.313310 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.313294    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfe8c234-c443-44a6-8420-9566768d6d02-istio-podinfo\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9\" (UID: \"bfe8c234-c443-44a6-8420-9566768d6d02\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9"
Oct 30 04:59:07.328972 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.328954    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfe8c234-c443-44a6-8420-9566768d6d02-istio-token\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9\" (UID: \"bfe8c234-c443-44a6-8420-9566768d6d02\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9"
Oct 30 04:59:07.329282 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.329261    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk8kc\" (UniqueName: \"kubernetes.io/projected/bfe8c234-c443-44a6-8420-9566768d6d02-kube-api-access-pk8kc\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9\" (UID: \"bfe8c234-c443-44a6-8420-9566768d6d02\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9"
Oct 30 04:59:07.448601 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.448571    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9"
Oct 30 04:59:07.622520 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.622492    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw"]
Oct 30 04:59:07.623778 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:59:07.623754    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cdfbc2d_3a92_4782_89cb_886f47416486.slice/crio-9cd400d310cf1ce3c36d2f7c15458dc176142622d03b45317d5243364c325ed0 WatchSource:0}: Error finding container 9cd400d310cf1ce3c36d2f7c15458dc176142622d03b45317d5243364c325ed0: Status 404 returned error can't find the container with id 9cd400d310cf1ce3c36d2f7c15458dc176142622d03b45317d5243364c325ed0
Oct 30 04:59:07.626170 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.626129    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 04:59:07.626261 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.626204    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 04:59:07.626261 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.626231    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 04:59:07.751138 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.751098    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9"]
Oct 30 04:59:07.755487 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:59:07.755455    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfe8c234_c443_44a6_8420_9566768d6d02.slice/crio-0610b4c9d3eb5f575067aed346d43cf749205e1dbe2edfb78fbbfe3acbb72f06 WatchSource:0}: Error finding container 0610b4c9d3eb5f575067aed346d43cf749205e1dbe2edfb78fbbfe3acbb72f06: Status 404 returned error can't find the container with id 0610b4c9d3eb5f575067aed346d43cf749205e1dbe2edfb78fbbfe3acbb72f06
Oct 30 04:59:07.758312 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.758272    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 04:59:07.758387 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.758362    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 04:59:07.758431 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.758404    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 04:59:07.944997 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.944881    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9" event={"ID":"bfe8c234-c443-44a6-8420-9566768d6d02","Type":"ContainerStarted","Data":"2e2c53a2338a2d2dbc3f8266d0fce15a8e52a00618606f23f455b15c420cc0cd"}
Oct 30 04:59:07.944997 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.944960    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9" event={"ID":"bfe8c234-c443-44a6-8420-9566768d6d02","Type":"ContainerStarted","Data":"0610b4c9d3eb5f575067aed346d43cf749205e1dbe2edfb78fbbfe3acbb72f06"}
Oct 30 04:59:07.947469 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.947383    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw" event={"ID":"8cdfbc2d-3a92-4782-89cb-886f47416486","Type":"ContainerStarted","Data":"d9adc65b9f0cc67eb73c801f80309adff85411bda04fb280a41252f6461875db"}
Oct 30 04:59:07.947469 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.947438    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw" event={"ID":"8cdfbc2d-3a92-4782-89cb-886f47416486","Type":"ContainerStarted","Data":"9cd400d310cf1ce3c36d2f7c15458dc176142622d03b45317d5243364c325ed0"}
Oct 30 04:59:07.984577 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:07.984524    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9" podStartSLOduration=0.984509377 podStartE2EDuration="984.509377ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:59:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:59:07.983503284 +0000 UTC m=+877.207323210" watchObservedRunningTime="2026-10-30 04:59:07.984509377 +0000 UTC m=+877.208329278"
Oct 30 04:59:08.024421 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:08.024376    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw" podStartSLOduration=2.024361759 podStartE2EDuration="2.024361759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:59:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:59:08.022864724 +0000 UTC m=+877.246684627" watchObservedRunningTime="2026-10-30 04:59:08.024361759 +0000 UTC m=+877.248181657"
Oct 30 04:59:08.318637 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:08.316312    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw"
Oct 30 04:59:08.318961 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:08.318934    5898 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 04:59:08.319130 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:08.319104    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw" podUID="8cdfbc2d-3a92-4782-89cb-886f47416486" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 04:59:08.449635 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:08.449604    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9"
Oct 30 04:59:08.451059 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:08.451037    5898 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 04:59:08.451184 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:08.451080    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9" podUID="bfe8c234-c443-44a6-8420-9566768d6d02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 04:59:08.743870 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:08.743830    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:59:09.315926 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:09.315325    5898 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 04:59:09.315926 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:09.315403    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw" podUID="8cdfbc2d-3a92-4782-89cb-886f47416486" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 04:59:09.449539 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:09.449507    5898 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 04:59:09.449767 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:09.449570    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9" podUID="bfe8c234-c443-44a6-8420-9566768d6d02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 04:59:10.313303 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:10.313265    5898 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 04:59:10.313620 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:10.313590    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw" podUID="8cdfbc2d-3a92-4782-89cb-886f47416486" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 04:59:10.433411 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:10.433362    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:59:10.449560 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:10.449526    5898 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 04:59:10.449843 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:10.449813    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9" podUID="bfe8c234-c443-44a6-8420-9566768d6d02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 04:59:10.542177 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:10.542145    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:59:11.317925 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:11.317239    5898 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 04:59:11.317925 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:11.317299    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw" podUID="8cdfbc2d-3a92-4782-89cb-886f47416486" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 04:59:11.449061 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:11.449030    5898 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 04:59:11.449455 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:11.449077    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9" podUID="bfe8c234-c443-44a6-8420-9566768d6d02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 04:59:12.116892 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:12.116858    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:59:12.317014 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:12.316373    5898 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 04:59:12.317014 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:12.316432    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw" podUID="8cdfbc2d-3a92-4782-89cb-886f47416486" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 04:59:12.449981 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:12.449374    5898 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 04:59:12.449981 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:12.449431    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9" podUID="bfe8c234-c443-44a6-8420-9566768d6d02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 04:59:13.314925 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:13.314242    5898 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 04:59:13.314925 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:13.314305    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw" podUID="8cdfbc2d-3a92-4782-89cb-886f47416486" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 04:59:13.449641 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:13.449593    5898 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 04:59:13.449819 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:13.449664    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9" podUID="bfe8c234-c443-44a6-8420-9566768d6d02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 04:59:14.316925 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:14.313305    5898 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 04:59:14.316925 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:14.313375    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw" podUID="8cdfbc2d-3a92-4782-89cb-886f47416486" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 04:59:14.404613 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:14.404568    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:59:14.925404 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:14.925367    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9"
Oct 30 04:59:14.925809 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:14.925779    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9"
Oct 30 04:59:14.927122 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:14.927098    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9"
Oct 30 04:59:15.318377 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:15.318286    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw"
Oct 30 04:59:15.318859 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:15.318632    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw"
Oct 30 04:59:15.319420 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:15.319401    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw"
Oct 30 04:59:16.683194 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:16.683150    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:59:18.302941 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:18.302912    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:59:19.963495 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:19.963465    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:59:21.671439 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:21.671404    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99"]
Oct 30 04:59:21.675115 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:21.675095    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99"
Oct 30 04:59:21.677960 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:21.677934    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:59:21.707046 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:21.707016    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-b21507-openshift-default-dockercfg-q4ptg\""
Oct 30 04:59:21.709617 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:21.709586    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99"]
Oct 30 04:59:21.823194 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:21.823159    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2421f64-3c01-4dbc-83b9-51d8a29b3e9c-credential-socket\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99\" (UID: \"b2421f64-3c01-4dbc-83b9-51d8a29b3e9c\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99"
Oct 30 04:59:21.823361 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:21.823259    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2421f64-3c01-4dbc-83b9-51d8a29b3e9c-istio-token\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99\" (UID: \"b2421f64-3c01-4dbc-83b9-51d8a29b3e9c\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99"
Oct 30 04:59:21.823361 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:21.823289    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prgk5\" (UniqueName: \"kubernetes.io/projected/b2421f64-3c01-4dbc-83b9-51d8a29b3e9c-kube-api-access-prgk5\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99\" (UID: \"b2421f64-3c01-4dbc-83b9-51d8a29b3e9c\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99"
Oct 30 04:59:21.823361 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:21.823316    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2421f64-3c01-4dbc-83b9-51d8a29b3e9c-workload-socket\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99\" (UID: \"b2421f64-3c01-4dbc-83b9-51d8a29b3e9c\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99"
Oct 30 04:59:21.823361 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:21.823340    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2421f64-3c01-4dbc-83b9-51d8a29b3e9c-workload-certs\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99\" (UID: \"b2421f64-3c01-4dbc-83b9-51d8a29b3e9c\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99"
Oct 30 04:59:21.823361 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:21.823357    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2421f64-3c01-4dbc-83b9-51d8a29b3e9c-istio-envoy\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99\" (UID: \"b2421f64-3c01-4dbc-83b9-51d8a29b3e9c\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99"
Oct 30 04:59:21.823524 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:21.823443    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2421f64-3c01-4dbc-83b9-51d8a29b3e9c-istio-data\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99\" (UID: \"b2421f64-3c01-4dbc-83b9-51d8a29b3e9c\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99"
Oct 30 04:59:21.823524 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:21.823473    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2421f64-3c01-4dbc-83b9-51d8a29b3e9c-istio-podinfo\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99\" (UID: \"b2421f64-3c01-4dbc-83b9-51d8a29b3e9c\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99"
Oct 30 04:59:21.823524 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:21.823509    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2421f64-3c01-4dbc-83b9-51d8a29b3e9c-istiod-ca-cert\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99\" (UID: \"b2421f64-3c01-4dbc-83b9-51d8a29b3e9c\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99"
Oct 30 04:59:21.924256 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:21.924181    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2421f64-3c01-4dbc-83b9-51d8a29b3e9c-istio-token\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99\" (UID: \"b2421f64-3c01-4dbc-83b9-51d8a29b3e9c\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99"
Oct 30 04:59:21.924256 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:21.924216    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prgk5\" (UniqueName: \"kubernetes.io/projected/b2421f64-3c01-4dbc-83b9-51d8a29b3e9c-kube-api-access-prgk5\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99\" (UID: \"b2421f64-3c01-4dbc-83b9-51d8a29b3e9c\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99"
Oct 30 04:59:21.924256 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:21.924239    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2421f64-3c01-4dbc-83b9-51d8a29b3e9c-workload-socket\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99\" (UID: \"b2421f64-3c01-4dbc-83b9-51d8a29b3e9c\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99"
Oct 30 04:59:21.924478 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:21.924272    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2421f64-3c01-4dbc-83b9-51d8a29b3e9c-workload-certs\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99\" (UID: \"b2421f64-3c01-4dbc-83b9-51d8a29b3e9c\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99"
Oct 30 04:59:21.924478 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:21.924297    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2421f64-3c01-4dbc-83b9-51d8a29b3e9c-istio-envoy\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99\" (UID: \"b2421f64-3c01-4dbc-83b9-51d8a29b3e9c\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99"
Oct 30 04:59:21.924478 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:21.924377    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2421f64-3c01-4dbc-83b9-51d8a29b3e9c-istio-data\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99\" (UID: \"b2421f64-3c01-4dbc-83b9-51d8a29b3e9c\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99"
Oct 30 04:59:21.924478 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:21.924412    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2421f64-3c01-4dbc-83b9-51d8a29b3e9c-istio-podinfo\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99\" (UID: \"b2421f64-3c01-4dbc-83b9-51d8a29b3e9c\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99"
Oct 30 04:59:21.924478 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:21.924461    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2421f64-3c01-4dbc-83b9-51d8a29b3e9c-istiod-ca-cert\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99\" (UID: \"b2421f64-3c01-4dbc-83b9-51d8a29b3e9c\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99"
Oct 30 04:59:21.924704 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:21.924532    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2421f64-3c01-4dbc-83b9-51d8a29b3e9c-credential-socket\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99\" (UID: \"b2421f64-3c01-4dbc-83b9-51d8a29b3e9c\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99"
Oct 30 04:59:21.924780 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:21.924756    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2421f64-3c01-4dbc-83b9-51d8a29b3e9c-workload-certs\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99\" (UID: \"b2421f64-3c01-4dbc-83b9-51d8a29b3e9c\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99"
Oct 30 04:59:21.924831 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:21.924785    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2421f64-3c01-4dbc-83b9-51d8a29b3e9c-workload-socket\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99\" (UID: \"b2421f64-3c01-4dbc-83b9-51d8a29b3e9c\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99"
Oct 30 04:59:21.925025 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:21.925002    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2421f64-3c01-4dbc-83b9-51d8a29b3e9c-credential-socket\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99\" (UID: \"b2421f64-3c01-4dbc-83b9-51d8a29b3e9c\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99"
Oct 30 04:59:21.925114 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:21.925018    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2421f64-3c01-4dbc-83b9-51d8a29b3e9c-istio-data\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99\" (UID: \"b2421f64-3c01-4dbc-83b9-51d8a29b3e9c\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99"
Oct 30 04:59:21.925253 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:21.925237    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2421f64-3c01-4dbc-83b9-51d8a29b3e9c-istiod-ca-cert\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99\" (UID: \"b2421f64-3c01-4dbc-83b9-51d8a29b3e9c\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99"
Oct 30 04:59:21.926821 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:21.926794    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2421f64-3c01-4dbc-83b9-51d8a29b3e9c-istio-envoy\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99\" (UID: \"b2421f64-3c01-4dbc-83b9-51d8a29b3e9c\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99"
Oct 30 04:59:21.926939 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:21.926924    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2421f64-3c01-4dbc-83b9-51d8a29b3e9c-istio-podinfo\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99\" (UID: \"b2421f64-3c01-4dbc-83b9-51d8a29b3e9c\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99"
Oct 30 04:59:21.942135 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:21.942107    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2421f64-3c01-4dbc-83b9-51d8a29b3e9c-istio-token\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99\" (UID: \"b2421f64-3c01-4dbc-83b9-51d8a29b3e9c\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99"
Oct 30 04:59:21.942229 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:21.942183    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prgk5\" (UniqueName: \"kubernetes.io/projected/b2421f64-3c01-4dbc-83b9-51d8a29b3e9c-kube-api-access-prgk5\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99\" (UID: \"b2421f64-3c01-4dbc-83b9-51d8a29b3e9c\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99"
Oct 30 04:59:21.989049 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:21.989026    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99"
Oct 30 04:59:22.300140 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:22.300110    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99"]
Oct 30 04:59:22.301231 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:59:22.301203    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2421f64_3c01_4dbc_83b9_51d8a29b3e9c.slice/crio-3f102e477b5fcae6a93c9e3ef275633a3ac084660d2f1b2327735b7e1471ecdd WatchSource:0}: Error finding container 3f102e477b5fcae6a93c9e3ef275633a3ac084660d2f1b2327735b7e1471ecdd: Status 404 returned error can't find the container with id 3f102e477b5fcae6a93c9e3ef275633a3ac084660d2f1b2327735b7e1471ecdd
Oct 30 04:59:22.303598 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:22.303565    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 04:59:22.303711 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:22.303652    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 04:59:22.303711 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:22.303696    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 04:59:23.065615 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:23.065576    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99" event={"ID":"b2421f64-3c01-4dbc-83b9-51d8a29b3e9c","Type":"ContainerStarted","Data":"ecb89465a459c6cd8b0072840729601554c930ef82c04372e43aaf70d3497dda"}
Oct 30 04:59:23.066132 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:23.065635    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99" event={"ID":"b2421f64-3c01-4dbc-83b9-51d8a29b3e9c","Type":"ContainerStarted","Data":"3f102e477b5fcae6a93c9e3ef275633a3ac084660d2f1b2327735b7e1471ecdd"}
Oct 30 04:59:23.454613 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:23.454580    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:59:23.993247 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:23.993212    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99"
Oct 30 04:59:23.995649 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:23.995611    5898 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 04:59:23.995856 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:23.995829    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99" podUID="b2421f64-3c01-4dbc-83b9-51d8a29b3e9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 04:59:24.991481 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:24.991445    5898 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 04:59:24.992117 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:24.992084    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99" podUID="b2421f64-3c01-4dbc-83b9-51d8a29b3e9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 04:59:25.469729 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:25.469697    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:59:25.995925 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:25.995050    5898 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 04:59:25.995925 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:25.995144    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99" podUID="b2421f64-3c01-4dbc-83b9-51d8a29b3e9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 04:59:26.505865 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:26.505804    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99" podStartSLOduration=5.505783622 podStartE2EDuration="5.505783622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:59:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:59:23.137825714 +0000 UTC m=+892.361645614" watchObservedRunningTime="2026-10-30 04:59:26.505783622 +0000 UTC m=+895.729603526"
Oct 30 04:59:26.506965 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:26.506938    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48"]
Oct 30 04:59:26.511694 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:26.511673    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48"
Oct 30 04:59:26.532608 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:26.532583    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48"]
Oct 30 04:59:26.597924 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:26.597537    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66f813c3-c7c9-452d-839d-4710108f03c4-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-k9r48\" (UID: \"66f813c3-c7c9-452d-839d-4710108f03c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48"
Oct 30 04:59:26.597924 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:26.597602    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66f813c3-c7c9-452d-839d-4710108f03c4-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-k9r48\" (UID: \"66f813c3-c7c9-452d-839d-4710108f03c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48"
Oct 30 04:59:26.597924 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:26.597632    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g55g\" (UniqueName: \"kubernetes.io/projected/66f813c3-c7c9-452d-839d-4710108f03c4-kube-api-access-4g55g\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-k9r48\" (UID: \"66f813c3-c7c9-452d-839d-4710108f03c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48"
Oct 30 04:59:26.597924 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:26.597669    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66f813c3-c7c9-452d-839d-4710108f03c4-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-k9r48\" (UID: \"66f813c3-c7c9-452d-839d-4710108f03c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48"
Oct 30 04:59:26.597924 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:26.597710    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66f813c3-c7c9-452d-839d-4710108f03c4-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-k9r48\" (UID: \"66f813c3-c7c9-452d-839d-4710108f03c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48"
Oct 30 04:59:26.597924 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:26.597787    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66f813c3-c7c9-452d-839d-4710108f03c4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-k9r48\" (UID: \"66f813c3-c7c9-452d-839d-4710108f03c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48"
Oct 30 04:59:26.597924 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:26.597817    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66f813c3-c7c9-452d-839d-4710108f03c4-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-k9r48\" (UID: \"66f813c3-c7c9-452d-839d-4710108f03c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48"
Oct 30 04:59:26.597924 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:26.597860    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66f813c3-c7c9-452d-839d-4710108f03c4-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-k9r48\" (UID: \"66f813c3-c7c9-452d-839d-4710108f03c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48"
Oct 30 04:59:26.597924 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:26.597913    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66f813c3-c7c9-452d-839d-4710108f03c4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-k9r48\" (UID: \"66f813c3-c7c9-452d-839d-4710108f03c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48"
Oct 30 04:59:26.705453 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:26.705414    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66f813c3-c7c9-452d-839d-4710108f03c4-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-k9r48\" (UID: \"66f813c3-c7c9-452d-839d-4710108f03c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48"
Oct 30 04:59:26.705453 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:26.705460    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66f813c3-c7c9-452d-839d-4710108f03c4-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-k9r48\" (UID: \"66f813c3-c7c9-452d-839d-4710108f03c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48"
Oct 30 04:59:26.705711 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:26.705477    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g55g\" (UniqueName: \"kubernetes.io/projected/66f813c3-c7c9-452d-839d-4710108f03c4-kube-api-access-4g55g\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-k9r48\" (UID: \"66f813c3-c7c9-452d-839d-4710108f03c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48"
Oct 30 04:59:26.705711 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:26.705505    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66f813c3-c7c9-452d-839d-4710108f03c4-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-k9r48\" (UID: \"66f813c3-c7c9-452d-839d-4710108f03c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48"
Oct 30 04:59:26.705711 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:26.705546    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66f813c3-c7c9-452d-839d-4710108f03c4-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-k9r48\" (UID: \"66f813c3-c7c9-452d-839d-4710108f03c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48"
Oct 30 04:59:26.705711 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:26.705621    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66f813c3-c7c9-452d-839d-4710108f03c4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-k9r48\" (UID: \"66f813c3-c7c9-452d-839d-4710108f03c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48"
Oct 30 04:59:26.705711 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:26.705648    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66f813c3-c7c9-452d-839d-4710108f03c4-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-k9r48\" (UID: \"66f813c3-c7c9-452d-839d-4710108f03c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48"
Oct 30 04:59:26.705711 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:26.705689    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66f813c3-c7c9-452d-839d-4710108f03c4-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-k9r48\" (UID: \"66f813c3-c7c9-452d-839d-4710108f03c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48"
Oct 30 04:59:26.706055 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:26.705725    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66f813c3-c7c9-452d-839d-4710108f03c4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-k9r48\" (UID: \"66f813c3-c7c9-452d-839d-4710108f03c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48"
Oct 30 04:59:26.706055 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:26.705880    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66f813c3-c7c9-452d-839d-4710108f03c4-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-k9r48\" (UID: \"66f813c3-c7c9-452d-839d-4710108f03c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48"
Oct 30 04:59:26.706055 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:26.705884    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66f813c3-c7c9-452d-839d-4710108f03c4-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-k9r48\" (UID: \"66f813c3-c7c9-452d-839d-4710108f03c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48"
Oct 30 04:59:26.706228 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:26.706121    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66f813c3-c7c9-452d-839d-4710108f03c4-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-k9r48\" (UID: \"66f813c3-c7c9-452d-839d-4710108f03c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48"
Oct 30 04:59:26.706287 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:26.706228    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66f813c3-c7c9-452d-839d-4710108f03c4-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-k9r48\" (UID: \"66f813c3-c7c9-452d-839d-4710108f03c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48"
Oct 30 04:59:26.706500 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:26.706474    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66f813c3-c7c9-452d-839d-4710108f03c4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-k9r48\" (UID: \"66f813c3-c7c9-452d-839d-4710108f03c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48"
Oct 30 04:59:26.708291 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:26.708264    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66f813c3-c7c9-452d-839d-4710108f03c4-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-k9r48\" (UID: \"66f813c3-c7c9-452d-839d-4710108f03c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48"
Oct 30 04:59:26.708485 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:26.708468    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66f813c3-c7c9-452d-839d-4710108f03c4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-k9r48\" (UID: \"66f813c3-c7c9-452d-839d-4710108f03c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48"
Oct 30 04:59:26.723713 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:26.723691    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g55g\" (UniqueName: \"kubernetes.io/projected/66f813c3-c7c9-452d-839d-4710108f03c4-kube-api-access-4g55g\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-k9r48\" (UID: \"66f813c3-c7c9-452d-839d-4710108f03c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48"
Oct 30 04:59:26.723836 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:26.723773    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66f813c3-c7c9-452d-839d-4710108f03c4-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-k9r48\" (UID: \"66f813c3-c7c9-452d-839d-4710108f03c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48"
Oct 30 04:59:26.825345 ip-10-0-135-98 kubenswrapper[5898]: E1030 04:59:26.825311    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 04:59:26.834693 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:26.834670    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48"
Oct 30 04:59:26.997228 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:26.997111    5898 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 04:59:27.001104 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:27.001072    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99" podUID="b2421f64-3c01-4dbc-83b9-51d8a29b3e9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 04:59:27.172079 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:27.172046    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48"]
Oct 30 04:59:27.172280 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:59:27.172251    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66f813c3_c7c9_452d_839d_4710108f03c4.slice/crio-f3c5054b96800f0c624b96cdc2529f585f505185e34e1d20f720fd44b48cb7c4 WatchSource:0}: Error finding container f3c5054b96800f0c624b96cdc2529f585f505185e34e1d20f720fd44b48cb7c4: Status 404 returned error can't find the container with id f3c5054b96800f0c624b96cdc2529f585f505185e34e1d20f720fd44b48cb7c4
Oct 30 04:59:27.197924 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:27.192791    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 04:59:27.197924 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:27.192887    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 04:59:27.197924 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:27.192951    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 04:59:27.309417 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:27.309005    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:59:27.994926 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:27.991278    5898 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 04:59:27.994926 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:27.991839    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99" podUID="b2421f64-3c01-4dbc-83b9-51d8a29b3e9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 04:59:28.133550 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:28.133503    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48" event={"ID":"66f813c3-c7c9-452d-839d-4710108f03c4","Type":"ContainerStarted","Data":"3f2bbd9c32545fbf42a40c68f01534d2b515e7194ce1c896e7d5e310c3949e34"}
Oct 30 04:59:28.134089 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:28.133586    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48" event={"ID":"66f813c3-c7c9-452d-839d-4710108f03c4","Type":"ContainerStarted","Data":"f3c5054b96800f0c624b96cdc2529f585f505185e34e1d20f720fd44b48cb7c4"}
Oct 30 04:59:28.201922 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:28.200890    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48" podStartSLOduration=2.200865607 podStartE2EDuration="2.200865607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:59:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:59:28.191020715 +0000 UTC m=+897.414840616" watchObservedRunningTime="2026-10-30 04:59:28.200865607 +0000 UTC m=+897.424685511"
Oct 30 04:59:28.835201 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:28.835165    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48"
Oct 30 04:59:28.836738 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:28.836713    5898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 04:59:28.836843 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:28.836770    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48" podUID="66f813c3-c7c9-452d-839d-4710108f03c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 04:59:28.950214 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:28.950179    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 04:59:29.044939 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:29.044859    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:59:29.128581 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:29.128502    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99"
Oct 30 04:59:29.128972 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:29.128937    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99"
Oct 30 04:59:29.129999 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:29.129939    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99"
Oct 30 04:59:29.565629 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:29.565538    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6"]
Oct 30 04:59:29.569952 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:29.569926    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6"
Oct 30 04:59:29.586799 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:29.586774    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b21507\"/\"distinct-w0-b21507-kserve-self-signed-certs\""
Oct 30 04:59:29.608952 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:29.605064    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6"]
Oct 30 04:59:29.617500 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:29.617469    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:59:29.750321 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:29.750283    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b09804e3-8135-4f6d-ae34-c45d6da9728e-home\") pod \"distinct-w0-b21507-kserve-866698fd8b-hjkp6\" (UID: \"b09804e3-8135-4f6d-ae34-c45d6da9728e\") " pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6"
Oct 30 04:59:29.750321 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:29.750324    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b09804e3-8135-4f6d-ae34-c45d6da9728e-tls-certs\") pod \"distinct-w0-b21507-kserve-866698fd8b-hjkp6\" (UID: \"b09804e3-8135-4f6d-ae34-c45d6da9728e\") " pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6"
Oct 30 04:59:29.750552 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:29.750525    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b09804e3-8135-4f6d-ae34-c45d6da9728e-tmp-dir\") pod \"distinct-w0-b21507-kserve-866698fd8b-hjkp6\" (UID: \"b09804e3-8135-4f6d-ae34-c45d6da9728e\") " pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6"
Oct 30 04:59:29.750650 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:29.750634    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b09804e3-8135-4f6d-ae34-c45d6da9728e-model-cache\") pod \"distinct-w0-b21507-kserve-866698fd8b-hjkp6\" (UID: \"b09804e3-8135-4f6d-ae34-c45d6da9728e\") " pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6"
Oct 30 04:59:29.750701 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:29.750686    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2jn9\" (UniqueName: \"kubernetes.io/projected/b09804e3-8135-4f6d-ae34-c45d6da9728e-kube-api-access-m2jn9\") pod \"distinct-w0-b21507-kserve-866698fd8b-hjkp6\" (UID: \"b09804e3-8135-4f6d-ae34-c45d6da9728e\") " pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6"
Oct 30 04:59:29.750803 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:29.750785    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b09804e3-8135-4f6d-ae34-c45d6da9728e-dshm\") pod \"distinct-w0-b21507-kserve-866698fd8b-hjkp6\" (UID: \"b09804e3-8135-4f6d-ae34-c45d6da9728e\") " pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6"
Oct 30 04:59:29.835502 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:29.835464    5898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 04:59:29.835792 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:29.835763    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48" podUID="66f813c3-c7c9-452d-839d-4710108f03c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 04:59:29.853149 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:29.853110    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b09804e3-8135-4f6d-ae34-c45d6da9728e-tmp-dir\") pod \"distinct-w0-b21507-kserve-866698fd8b-hjkp6\" (UID: \"b09804e3-8135-4f6d-ae34-c45d6da9728e\") " pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6"
Oct 30 04:59:29.853441 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:29.853418    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b09804e3-8135-4f6d-ae34-c45d6da9728e-model-cache\") pod \"distinct-w0-b21507-kserve-866698fd8b-hjkp6\" (UID: \"b09804e3-8135-4f6d-ae34-c45d6da9728e\") " pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6"
Oct 30 04:59:29.853579 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:29.853562    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2jn9\" (UniqueName: \"kubernetes.io/projected/b09804e3-8135-4f6d-ae34-c45d6da9728e-kube-api-access-m2jn9\") pod \"distinct-w0-b21507-kserve-866698fd8b-hjkp6\" (UID: \"b09804e3-8135-4f6d-ae34-c45d6da9728e\") " pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6"
Oct 30 04:59:29.853716 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:29.853702    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b09804e3-8135-4f6d-ae34-c45d6da9728e-dshm\") pod \"distinct-w0-b21507-kserve-866698fd8b-hjkp6\" (UID: \"b09804e3-8135-4f6d-ae34-c45d6da9728e\") " pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6"
Oct 30 04:59:29.853872 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:29.853857    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b09804e3-8135-4f6d-ae34-c45d6da9728e-home\") pod \"distinct-w0-b21507-kserve-866698fd8b-hjkp6\" (UID: \"b09804e3-8135-4f6d-ae34-c45d6da9728e\") " pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6"
Oct 30 04:59:29.853994 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:29.853979    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b09804e3-8135-4f6d-ae34-c45d6da9728e-tls-certs\") pod \"distinct-w0-b21507-kserve-866698fd8b-hjkp6\" (UID: \"b09804e3-8135-4f6d-ae34-c45d6da9728e\") " pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6"
Oct 30 04:59:29.854805 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:29.854777    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b09804e3-8135-4f6d-ae34-c45d6da9728e-tmp-dir\") pod \"distinct-w0-b21507-kserve-866698fd8b-hjkp6\" (UID: \"b09804e3-8135-4f6d-ae34-c45d6da9728e\") " pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6"
Oct 30 04:59:29.855077 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:29.855057    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b09804e3-8135-4f6d-ae34-c45d6da9728e-model-cache\") pod \"distinct-w0-b21507-kserve-866698fd8b-hjkp6\" (UID: \"b09804e3-8135-4f6d-ae34-c45d6da9728e\") " pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6"
Oct 30 04:59:29.855932 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:29.855876    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b09804e3-8135-4f6d-ae34-c45d6da9728e-home\") pod \"distinct-w0-b21507-kserve-866698fd8b-hjkp6\" (UID: \"b09804e3-8135-4f6d-ae34-c45d6da9728e\") " pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6"
Oct 30 04:59:29.868745 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:29.868620    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b09804e3-8135-4f6d-ae34-c45d6da9728e-dshm\") pod \"distinct-w0-b21507-kserve-866698fd8b-hjkp6\" (UID: \"b09804e3-8135-4f6d-ae34-c45d6da9728e\") " pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6"
Oct 30 04:59:29.868745 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:29.868633    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b09804e3-8135-4f6d-ae34-c45d6da9728e-tls-certs\") pod \"distinct-w0-b21507-kserve-866698fd8b-hjkp6\" (UID: \"b09804e3-8135-4f6d-ae34-c45d6da9728e\") " pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6"
Oct 30 04:59:29.879625 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:29.879584    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2jn9\" (UniqueName: \"kubernetes.io/projected/b09804e3-8135-4f6d-ae34-c45d6da9728e-kube-api-access-m2jn9\") pod \"distinct-w0-b21507-kserve-866698fd8b-hjkp6\" (UID: \"b09804e3-8135-4f6d-ae34-c45d6da9728e\") " pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6"
Oct 30 04:59:29.918381 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:29.918348    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6"
Oct 30 04:59:30.330172 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:30.329877    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6"]
Oct 30 04:59:30.836925 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:30.836364    5898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 04:59:30.836925 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:30.836423    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48" podUID="66f813c3-c7c9-452d-839d-4710108f03c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 04:59:31.187006 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:31.179112    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6" event={"ID":"b09804e3-8135-4f6d-ae34-c45d6da9728e","Type":"ContainerStarted","Data":"28741bfcd2964321cba1ee5775c787f0ab4c32774a47338b54efd3b2d9c05fc4"}
Oct 30 04:59:31.187006 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:31.179171    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6" event={"ID":"b09804e3-8135-4f6d-ae34-c45d6da9728e","Type":"ContainerStarted","Data":"ff76c08a1188f348b2c4e99fbcc765e4d962457401ce334bc583ad4fb1aee2fe"}
Oct 30 04:59:31.399924 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:31.396435    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:59:31.835677 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:31.835645    5898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 04:59:31.835953 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:31.835913    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48" podUID="66f813c3-c7c9-452d-839d-4710108f03c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 04:59:31.918645 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:31.918611    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6"
Oct 30 04:59:32.835758 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:32.835720    5898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 04:59:32.835953 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:32.835771    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48" podUID="66f813c3-c7c9-452d-839d-4710108f03c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 04:59:33.223786 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:33.223702    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:59:33.835242 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:33.835199    5898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 04:59:33.835421 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:33.835274    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48" podUID="66f813c3-c7c9-452d-839d-4710108f03c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 04:59:34.274287 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:34.274170    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:59:34.835530 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:34.835490    5898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 04:59:34.835797 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:34.835572    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48" podUID="66f813c3-c7c9-452d-839d-4710108f03c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 04:59:34.969682 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:34.969650    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:59:36.005587 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:36.005554    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48"
Oct 30 04:59:36.006145 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:36.006121    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48"
Oct 30 04:59:36.006791 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:36.006770    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48"
Oct 30 04:59:36.057141 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:36.057083    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6" podStartSLOduration=7.057068551 podStartE2EDuration="7.057068551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:59:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:59:31.220511875 +0000 UTC m=+900.444331804" watchObservedRunningTime="2026-10-30 04:59:36.057068551 +0000 UTC m=+905.280888451"
Oct 30 04:59:36.663940 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:36.663889    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:59:36.749660 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:36.749622    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:59:36.788510 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:36.788470    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7"]
Oct 30 04:59:36.793939 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:36.793884    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7"
Oct 30 04:59:36.812058 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:36.812033    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-889ca9-openshift-default-dockercfg-t56wr\""
Oct 30 04:59:36.826375 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:36.826340    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7"]
Oct 30 04:59:36.864287 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:36.864245    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e9e9b05-1135-4cde-8a29-42c7291befe3-istiod-ca-cert\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7\" (UID: \"4e9e9b05-1135-4cde-8a29-42c7291befe3\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7"
Oct 30 04:59:36.864657 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:36.864630    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e9e9b05-1135-4cde-8a29-42c7291befe3-workload-socket\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7\" (UID: \"4e9e9b05-1135-4cde-8a29-42c7291befe3\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7"
Oct 30 04:59:36.864848 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:36.864829    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e9e9b05-1135-4cde-8a29-42c7291befe3-istio-envoy\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7\" (UID: \"4e9e9b05-1135-4cde-8a29-42c7291befe3\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7"
Oct 30 04:59:36.864998 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:36.864980    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e9e9b05-1135-4cde-8a29-42c7291befe3-istio-data\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7\" (UID: \"4e9e9b05-1135-4cde-8a29-42c7291befe3\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7"
Oct 30 04:59:36.865161 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:36.865144    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z42bm\" (UniqueName: \"kubernetes.io/projected/4e9e9b05-1135-4cde-8a29-42c7291befe3-kube-api-access-z42bm\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7\" (UID: \"4e9e9b05-1135-4cde-8a29-42c7291befe3\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7"
Oct 30 04:59:36.865321 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:36.865304    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e9e9b05-1135-4cde-8a29-42c7291befe3-workload-certs\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7\" (UID: \"4e9e9b05-1135-4cde-8a29-42c7291befe3\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7"
Oct 30 04:59:36.865458 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:36.865442    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e9e9b05-1135-4cde-8a29-42c7291befe3-istio-token\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7\" (UID: \"4e9e9b05-1135-4cde-8a29-42c7291befe3\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7"
Oct 30 04:59:36.865581 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:36.865564    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e9e9b05-1135-4cde-8a29-42c7291befe3-credential-socket\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7\" (UID: \"4e9e9b05-1135-4cde-8a29-42c7291befe3\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7"
Oct 30 04:59:36.865726 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:36.865711    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e9e9b05-1135-4cde-8a29-42c7291befe3-istio-podinfo\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7\" (UID: \"4e9e9b05-1135-4cde-8a29-42c7291befe3\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7"
Oct 30 04:59:36.967639 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:36.967524    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e9e9b05-1135-4cde-8a29-42c7291befe3-workload-certs\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7\" (UID: \"4e9e9b05-1135-4cde-8a29-42c7291befe3\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7"
Oct 30 04:59:36.967639 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:36.967597    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e9e9b05-1135-4cde-8a29-42c7291befe3-istio-token\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7\" (UID: \"4e9e9b05-1135-4cde-8a29-42c7291befe3\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7"
Oct 30 04:59:36.967891 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:36.967719    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e9e9b05-1135-4cde-8a29-42c7291befe3-credential-socket\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7\" (UID: \"4e9e9b05-1135-4cde-8a29-42c7291befe3\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7"
Oct 30 04:59:36.967891 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:36.967793    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e9e9b05-1135-4cde-8a29-42c7291befe3-istio-podinfo\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7\" (UID: \"4e9e9b05-1135-4cde-8a29-42c7291befe3\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7"
Oct 30 04:59:36.968041 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:36.967937    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e9e9b05-1135-4cde-8a29-42c7291befe3-istiod-ca-cert\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7\" (UID: \"4e9e9b05-1135-4cde-8a29-42c7291befe3\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7"
Oct 30 04:59:36.968041 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:36.967990    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e9e9b05-1135-4cde-8a29-42c7291befe3-workload-socket\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7\" (UID: \"4e9e9b05-1135-4cde-8a29-42c7291befe3\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7"
Oct 30 04:59:36.968041 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:36.968016    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e9e9b05-1135-4cde-8a29-42c7291befe3-istio-envoy\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7\" (UID: \"4e9e9b05-1135-4cde-8a29-42c7291befe3\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7"
Oct 30 04:59:36.968041 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:36.968026    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e9e9b05-1135-4cde-8a29-42c7291befe3-workload-certs\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7\" (UID: \"4e9e9b05-1135-4cde-8a29-42c7291befe3\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7"
Oct 30 04:59:36.968246 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:36.968080    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e9e9b05-1135-4cde-8a29-42c7291befe3-istio-data\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7\" (UID: \"4e9e9b05-1135-4cde-8a29-42c7291befe3\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7"
Oct 30 04:59:36.968246 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:36.968138    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e9e9b05-1135-4cde-8a29-42c7291befe3-credential-socket\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7\" (UID: \"4e9e9b05-1135-4cde-8a29-42c7291befe3\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7"
Oct 30 04:59:36.968246 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:36.968149    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z42bm\" (UniqueName: \"kubernetes.io/projected/4e9e9b05-1135-4cde-8a29-42c7291befe3-kube-api-access-z42bm\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7\" (UID: \"4e9e9b05-1135-4cde-8a29-42c7291befe3\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7"
Oct 30 04:59:36.968486 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:36.968389    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e9e9b05-1135-4cde-8a29-42c7291befe3-istio-data\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7\" (UID: \"4e9e9b05-1135-4cde-8a29-42c7291befe3\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7"
Oct 30 04:59:36.968486 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:36.968457    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e9e9b05-1135-4cde-8a29-42c7291befe3-workload-socket\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7\" (UID: \"4e9e9b05-1135-4cde-8a29-42c7291befe3\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7"
Oct 30 04:59:36.968731 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:36.968706    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e9e9b05-1135-4cde-8a29-42c7291befe3-istiod-ca-cert\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7\" (UID: \"4e9e9b05-1135-4cde-8a29-42c7291befe3\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7"
Oct 30 04:59:36.970245 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:36.970216    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e9e9b05-1135-4cde-8a29-42c7291befe3-istio-envoy\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7\" (UID: \"4e9e9b05-1135-4cde-8a29-42c7291befe3\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7"
Oct 30 04:59:36.970380 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:36.970362    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e9e9b05-1135-4cde-8a29-42c7291befe3-istio-podinfo\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7\" (UID: \"4e9e9b05-1135-4cde-8a29-42c7291befe3\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7"
Oct 30 04:59:36.985953 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:36.985930    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e9e9b05-1135-4cde-8a29-42c7291befe3-istio-token\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7\" (UID: \"4e9e9b05-1135-4cde-8a29-42c7291befe3\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7"
Oct 30 04:59:36.986059 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:36.986007    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z42bm\" (UniqueName: \"kubernetes.io/projected/4e9e9b05-1135-4cde-8a29-42c7291befe3-kube-api-access-z42bm\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7\" (UID: \"4e9e9b05-1135-4cde-8a29-42c7291befe3\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7"
Oct 30 04:59:37.109890 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:37.109854    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7"
Oct 30 04:59:37.420267 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:37.420237    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7"]
Oct 30 04:59:37.420983 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:59:37.420950    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e9e9b05_1135_4cde_8a29_42c7291befe3.slice/crio-a0ace625ab33db4bfc95255dcc7d9e69b3b7d541c966b1a9076c23110dd592b2 WatchSource:0}: Error finding container a0ace625ab33db4bfc95255dcc7d9e69b3b7d541c966b1a9076c23110dd592b2: Status 404 returned error can't find the container with id a0ace625ab33db4bfc95255dcc7d9e69b3b7d541c966b1a9076c23110dd592b2
Oct 30 04:59:37.423384 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:37.423349    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 04:59:37.423474 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:37.423415    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 04:59:37.423474 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:37.423443    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 04:59:38.239137 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:38.239102    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7" event={"ID":"4e9e9b05-1135-4cde-8a29-42c7291befe3","Type":"ContainerStarted","Data":"f8752ce9d7ac11ac1013196c5b9ae5e8ab29ed555c946ffa0469a0382b95e89b"}
Oct 30 04:59:38.239595 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:38.239165    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7" event={"ID":"4e9e9b05-1135-4cde-8a29-42c7291befe3","Type":"ContainerStarted","Data":"a0ace625ab33db4bfc95255dcc7d9e69b3b7d541c966b1a9076c23110dd592b2"}
Oct 30 04:59:38.304063 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:38.303998    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7" podStartSLOduration=2.303979605 podStartE2EDuration="2.303979605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:59:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:59:38.299356578 +0000 UTC m=+907.523176508" watchObservedRunningTime="2026-10-30 04:59:38.303979605 +0000 UTC m=+907.527799505"
Oct 30 04:59:38.321028 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:38.320997    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:59:38.467110 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:38.467080    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:59:39.110872 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:39.110835    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7"
Oct 30 04:59:39.112580 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:39.112554    5898 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 04:59:39.112714 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:39.112602    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7" podUID="4e9e9b05-1135-4cde-8a29-42c7291befe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 04:59:39.918429 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:39.918394    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6"
Oct 30 04:59:39.931009 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:39.930985    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6"
Oct 30 04:59:40.110908 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:40.110862    5898 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 04:59:40.111080 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:40.110929    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7" podUID="4e9e9b05-1135-4cde-8a29-42c7291befe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 04:59:40.120708 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:40.120676    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:59:40.262159 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:40.262080    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6"
Oct 30 04:59:41.111303 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:41.111265    5898 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 04:59:41.111735 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:41.111324    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7" podUID="4e9e9b05-1135-4cde-8a29-42c7291befe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 04:59:42.110977 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:42.110345    5898 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 04:59:42.110977 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:42.110410    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7" podUID="4e9e9b05-1135-4cde-8a29-42c7291befe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 04:59:42.230179 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:42.230146    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:59:43.110408 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:43.110368    5898 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 04:59:43.110629 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:43.110463    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7" podUID="4e9e9b05-1135-4cde-8a29-42c7291befe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 04:59:43.935162 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:43.935128    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:59:44.110396 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:44.110359    5898 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 04:59:44.110610 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:44.110411    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7" podUID="4e9e9b05-1135-4cde-8a29-42c7291befe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 04:59:45.333226 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:45.333178    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7"
Oct 30 04:59:45.333779 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:45.333625    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7"
Oct 30 04:59:45.334612 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:45.334573    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7"
Oct 30 04:59:45.702613 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:45.702532    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:59:47.456328 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:47.456288    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:59:48.433255 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:48.433220    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:59:49.172885 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:49.172851    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:59:50.944676 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:50.944643    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:59:52.001950 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.001915    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf"]
Oct 30 04:59:52.005305 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.005285    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf"
Oct 30 04:59:52.034575 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.034548    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf"]
Oct 30 04:59:52.096717 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.096682    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l6v6\" (UniqueName: \"kubernetes.io/projected/b5d063ac-6da8-4217-bddd-82a05db7bc50-kube-api-access-7l6v6\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf\" (UID: \"b5d063ac-6da8-4217-bddd-82a05db7bc50\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf"
Oct 30 04:59:52.096849 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.096726    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5d063ac-6da8-4217-bddd-82a05db7bc50-istio-token\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf\" (UID: \"b5d063ac-6da8-4217-bddd-82a05db7bc50\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf"
Oct 30 04:59:52.096849 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.096824    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5d063ac-6da8-4217-bddd-82a05db7bc50-istiod-ca-cert\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf\" (UID: \"b5d063ac-6da8-4217-bddd-82a05db7bc50\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf"
Oct 30 04:59:52.096948 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.096854    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d063ac-6da8-4217-bddd-82a05db7bc50-credential-socket\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf\" (UID: \"b5d063ac-6da8-4217-bddd-82a05db7bc50\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf"
Oct 30 04:59:52.096984 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.096965    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5d063ac-6da8-4217-bddd-82a05db7bc50-istio-podinfo\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf\" (UID: \"b5d063ac-6da8-4217-bddd-82a05db7bc50\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf"
Oct 30 04:59:52.097037 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.097023    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5d063ac-6da8-4217-bddd-82a05db7bc50-istio-envoy\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf\" (UID: \"b5d063ac-6da8-4217-bddd-82a05db7bc50\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf"
Oct 30 04:59:52.097081 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.097070    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d063ac-6da8-4217-bddd-82a05db7bc50-workload-socket\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf\" (UID: \"b5d063ac-6da8-4217-bddd-82a05db7bc50\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf"
Oct 30 04:59:52.097125 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.097088    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5d063ac-6da8-4217-bddd-82a05db7bc50-workload-certs\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf\" (UID: \"b5d063ac-6da8-4217-bddd-82a05db7bc50\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf"
Oct 30 04:59:52.097186 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.097172    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5d063ac-6da8-4217-bddd-82a05db7bc50-istio-data\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf\" (UID: \"b5d063ac-6da8-4217-bddd-82a05db7bc50\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf"
Oct 30 04:59:52.198163 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.198134    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l6v6\" (UniqueName: \"kubernetes.io/projected/b5d063ac-6da8-4217-bddd-82a05db7bc50-kube-api-access-7l6v6\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf\" (UID: \"b5d063ac-6da8-4217-bddd-82a05db7bc50\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf"
Oct 30 04:59:52.198312 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.198184    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5d063ac-6da8-4217-bddd-82a05db7bc50-istio-token\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf\" (UID: \"b5d063ac-6da8-4217-bddd-82a05db7bc50\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf"
Oct 30 04:59:52.198312 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.198228    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5d063ac-6da8-4217-bddd-82a05db7bc50-istiod-ca-cert\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf\" (UID: \"b5d063ac-6da8-4217-bddd-82a05db7bc50\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf"
Oct 30 04:59:52.198312 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.198248    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d063ac-6da8-4217-bddd-82a05db7bc50-credential-socket\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf\" (UID: \"b5d063ac-6da8-4217-bddd-82a05db7bc50\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf"
Oct 30 04:59:52.198483 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.198462    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5d063ac-6da8-4217-bddd-82a05db7bc50-istio-podinfo\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf\" (UID: \"b5d063ac-6da8-4217-bddd-82a05db7bc50\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf"
Oct 30 04:59:52.198544 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.198526    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5d063ac-6da8-4217-bddd-82a05db7bc50-istio-envoy\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf\" (UID: \"b5d063ac-6da8-4217-bddd-82a05db7bc50\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf"
Oct 30 04:59:52.198600 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.198585    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d063ac-6da8-4217-bddd-82a05db7bc50-credential-socket\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf\" (UID: \"b5d063ac-6da8-4217-bddd-82a05db7bc50\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf"
Oct 30 04:59:52.198600 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.198591    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d063ac-6da8-4217-bddd-82a05db7bc50-workload-socket\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf\" (UID: \"b5d063ac-6da8-4217-bddd-82a05db7bc50\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf"
Oct 30 04:59:52.198704 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.198618    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5d063ac-6da8-4217-bddd-82a05db7bc50-workload-certs\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf\" (UID: \"b5d063ac-6da8-4217-bddd-82a05db7bc50\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf"
Oct 30 04:59:52.198704 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.198680    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5d063ac-6da8-4217-bddd-82a05db7bc50-istio-data\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf\" (UID: \"b5d063ac-6da8-4217-bddd-82a05db7bc50\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf"
Oct 30 04:59:52.198875 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.198846    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d063ac-6da8-4217-bddd-82a05db7bc50-workload-socket\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf\" (UID: \"b5d063ac-6da8-4217-bddd-82a05db7bc50\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf"
Oct 30 04:59:52.199036 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.198968    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5d063ac-6da8-4217-bddd-82a05db7bc50-istiod-ca-cert\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf\" (UID: \"b5d063ac-6da8-4217-bddd-82a05db7bc50\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf"
Oct 30 04:59:52.199116 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.199052    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5d063ac-6da8-4217-bddd-82a05db7bc50-istio-data\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf\" (UID: \"b5d063ac-6da8-4217-bddd-82a05db7bc50\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf"
Oct 30 04:59:52.199116 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.199089    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5d063ac-6da8-4217-bddd-82a05db7bc50-workload-certs\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf\" (UID: \"b5d063ac-6da8-4217-bddd-82a05db7bc50\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf"
Oct 30 04:59:52.200766 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.200741    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5d063ac-6da8-4217-bddd-82a05db7bc50-istio-envoy\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf\" (UID: \"b5d063ac-6da8-4217-bddd-82a05db7bc50\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf"
Oct 30 04:59:52.200893 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.200875    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5d063ac-6da8-4217-bddd-82a05db7bc50-istio-podinfo\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf\" (UID: \"b5d063ac-6da8-4217-bddd-82a05db7bc50\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf"
Oct 30 04:59:52.215993 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.215973    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l6v6\" (UniqueName: \"kubernetes.io/projected/b5d063ac-6da8-4217-bddd-82a05db7bc50-kube-api-access-7l6v6\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf\" (UID: \"b5d063ac-6da8-4217-bddd-82a05db7bc50\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf"
Oct 30 04:59:52.216114 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.216018    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5d063ac-6da8-4217-bddd-82a05db7bc50-istio-token\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf\" (UID: \"b5d063ac-6da8-4217-bddd-82a05db7bc50\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf"
Oct 30 04:59:52.272715 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.272657    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2"]
Oct 30 04:59:52.277554 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.277536    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2"
Oct 30 04:59:52.297713 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.297687    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2"]
Oct 30 04:59:52.320097 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.320073    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf"
Oct 30 04:59:52.402915 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.402859    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/983a192c-67a4-4684-86fc-6df70f38a1ed-istiod-ca-cert\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2\" (UID: \"983a192c-67a4-4684-86fc-6df70f38a1ed\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2"
Oct 30 04:59:52.403047 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.403034    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/983a192c-67a4-4684-86fc-6df70f38a1ed-workload-socket\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2\" (UID: \"983a192c-67a4-4684-86fc-6df70f38a1ed\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2"
Oct 30 04:59:52.403088 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.403053    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/983a192c-67a4-4684-86fc-6df70f38a1ed-istio-data\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2\" (UID: \"983a192c-67a4-4684-86fc-6df70f38a1ed\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2"
Oct 30 04:59:52.403124 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.403091    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/983a192c-67a4-4684-86fc-6df70f38a1ed-istio-podinfo\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2\" (UID: \"983a192c-67a4-4684-86fc-6df70f38a1ed\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2"
Oct 30 04:59:52.403212 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.403196    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/983a192c-67a4-4684-86fc-6df70f38a1ed-istio-envoy\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2\" (UID: \"983a192c-67a4-4684-86fc-6df70f38a1ed\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2"
Oct 30 04:59:52.403311 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.403286    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/983a192c-67a4-4684-86fc-6df70f38a1ed-workload-certs\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2\" (UID: \"983a192c-67a4-4684-86fc-6df70f38a1ed\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2"
Oct 30 04:59:52.403357 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.403343    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/983a192c-67a4-4684-86fc-6df70f38a1ed-istio-token\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2\" (UID: \"983a192c-67a4-4684-86fc-6df70f38a1ed\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2"
Oct 30 04:59:52.403447 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.403432    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq96x\" (UniqueName: \"kubernetes.io/projected/983a192c-67a4-4684-86fc-6df70f38a1ed-kube-api-access-wq96x\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2\" (UID: \"983a192c-67a4-4684-86fc-6df70f38a1ed\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2"
Oct 30 04:59:52.403518 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.403505    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/983a192c-67a4-4684-86fc-6df70f38a1ed-credential-socket\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2\" (UID: \"983a192c-67a4-4684-86fc-6df70f38a1ed\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2"
Oct 30 04:59:52.504128 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.504094    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/983a192c-67a4-4684-86fc-6df70f38a1ed-istiod-ca-cert\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2\" (UID: \"983a192c-67a4-4684-86fc-6df70f38a1ed\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2"
Oct 30 04:59:52.504284 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.504146    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/983a192c-67a4-4684-86fc-6df70f38a1ed-workload-socket\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2\" (UID: \"983a192c-67a4-4684-86fc-6df70f38a1ed\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2"
Oct 30 04:59:52.504284 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.504163    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/983a192c-67a4-4684-86fc-6df70f38a1ed-istio-data\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2\" (UID: \"983a192c-67a4-4684-86fc-6df70f38a1ed\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2"
Oct 30 04:59:52.504401 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.504277    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/983a192c-67a4-4684-86fc-6df70f38a1ed-istio-podinfo\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2\" (UID: \"983a192c-67a4-4684-86fc-6df70f38a1ed\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2"
Oct 30 04:59:52.504401 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.504327    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/983a192c-67a4-4684-86fc-6df70f38a1ed-istio-envoy\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2\" (UID: \"983a192c-67a4-4684-86fc-6df70f38a1ed\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2"
Oct 30 04:59:52.504401 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.504364    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/983a192c-67a4-4684-86fc-6df70f38a1ed-workload-certs\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2\" (UID: \"983a192c-67a4-4684-86fc-6df70f38a1ed\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2"
Oct 30 04:59:52.504566 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.504409    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/983a192c-67a4-4684-86fc-6df70f38a1ed-istio-token\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2\" (UID: \"983a192c-67a4-4684-86fc-6df70f38a1ed\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2"
Oct 30 04:59:52.504566 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.504489    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq96x\" (UniqueName: \"kubernetes.io/projected/983a192c-67a4-4684-86fc-6df70f38a1ed-kube-api-access-wq96x\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2\" (UID: \"983a192c-67a4-4684-86fc-6df70f38a1ed\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2"
Oct 30 04:59:52.504566 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.504540    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/983a192c-67a4-4684-86fc-6df70f38a1ed-istio-data\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2\" (UID: \"983a192c-67a4-4684-86fc-6df70f38a1ed\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2"
Oct 30 04:59:52.504566 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.504561    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/983a192c-67a4-4684-86fc-6df70f38a1ed-credential-socket\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2\" (UID: \"983a192c-67a4-4684-86fc-6df70f38a1ed\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2"
Oct 30 04:59:52.505242 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.505211    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/983a192c-67a4-4684-86fc-6df70f38a1ed-credential-socket\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2\" (UID: \"983a192c-67a4-4684-86fc-6df70f38a1ed\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2"
Oct 30 04:59:52.505383 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.505239    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/983a192c-67a4-4684-86fc-6df70f38a1ed-workload-certs\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2\" (UID: \"983a192c-67a4-4684-86fc-6df70f38a1ed\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2"
Oct 30 04:59:52.505383 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.505340    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/983a192c-67a4-4684-86fc-6df70f38a1ed-workload-socket\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2\" (UID: \"983a192c-67a4-4684-86fc-6df70f38a1ed\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2"
Oct 30 04:59:52.505557 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.505466    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/983a192c-67a4-4684-86fc-6df70f38a1ed-istiod-ca-cert\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2\" (UID: \"983a192c-67a4-4684-86fc-6df70f38a1ed\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2"
Oct 30 04:59:52.506718 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.506696    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/983a192c-67a4-4684-86fc-6df70f38a1ed-istio-envoy\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2\" (UID: \"983a192c-67a4-4684-86fc-6df70f38a1ed\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2"
Oct 30 04:59:52.506997 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.506979    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/983a192c-67a4-4684-86fc-6df70f38a1ed-istio-podinfo\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2\" (UID: \"983a192c-67a4-4684-86fc-6df70f38a1ed\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2"
Oct 30 04:59:52.522979 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.522927    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/983a192c-67a4-4684-86fc-6df70f38a1ed-istio-token\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2\" (UID: \"983a192c-67a4-4684-86fc-6df70f38a1ed\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2"
Oct 30 04:59:52.523180 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.523162    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq96x\" (UniqueName: \"kubernetes.io/projected/983a192c-67a4-4684-86fc-6df70f38a1ed-kube-api-access-wq96x\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2\" (UID: \"983a192c-67a4-4684-86fc-6df70f38a1ed\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2"
Oct 30 04:59:52.592557 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.592537    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2"
Oct 30 04:59:52.636578 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.634193    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf"]
Oct 30 04:59:52.642246 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.642211    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 04:59:52.642365 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.642300    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 04:59:52.642365 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.642340    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 04:59:52.743796 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.743749    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:59:52.924950 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.924921    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2"]
Oct 30 04:59:52.925587 ip-10-0-135-98 kubenswrapper[5898]: W1030 04:59:52.925560    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod983a192c_67a4_4684_86fc_6df70f38a1ed.slice/crio-4a6323685f3b09c1e0bbc9f253d767e484d8701eca5105ade44d9153c878a668 WatchSource:0}: Error finding container 4a6323685f3b09c1e0bbc9f253d767e484d8701eca5105ade44d9153c878a668: Status 404 returned error can't find the container with id 4a6323685f3b09c1e0bbc9f253d767e484d8701eca5105ade44d9153c878a668
Oct 30 04:59:52.927847 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.927815    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 04:59:52.927930 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.927876    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 04:59:52.927966 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:52.927938    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 04:59:53.336618 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:53.336584    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2" event={"ID":"983a192c-67a4-4684-86fc-6df70f38a1ed","Type":"ContainerStarted","Data":"490d7a49f4d76a3088a22e1fbc1a4ff8b3584250f52ebe3cb712e4b293542202"}
Oct 30 04:59:53.337055 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:53.336641    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2" event={"ID":"983a192c-67a4-4684-86fc-6df70f38a1ed","Type":"ContainerStarted","Data":"4a6323685f3b09c1e0bbc9f253d767e484d8701eca5105ade44d9153c878a668"}
Oct 30 04:59:53.338393 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:53.338364    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf" event={"ID":"b5d063ac-6da8-4217-bddd-82a05db7bc50","Type":"ContainerStarted","Data":"48e96e61aa3964057d215d81111d0e267b88a3bbafcd91aeef75ef3fb8105a71"}
Oct 30 04:59:53.338505 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:53.338419    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf" event={"ID":"b5d063ac-6da8-4217-bddd-82a05db7bc50","Type":"ContainerStarted","Data":"e51b3f4f591cf82a1daef302807e5eb76b78fef0ed16bceadb80a1441792e65b"}
Oct 30 04:59:53.379836 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:53.379791    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2" podStartSLOduration=1.3797790810000001 podStartE2EDuration="1.379779081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:59:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:59:53.378614336 +0000 UTC m=+922.602434271" watchObservedRunningTime="2026-10-30 04:59:53.379779081 +0000 UTC m=+922.603598982"
Oct 30 04:59:53.418622 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:53.418577    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf" podStartSLOduration=2.418560764 podStartE2EDuration="2.418560764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:59:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:59:53.417262989 +0000 UTC m=+922.641082915" watchObservedRunningTime="2026-10-30 04:59:53.418560764 +0000 UTC m=+922.642380665"
Oct 30 04:59:53.597984 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:53.594996    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2"
Oct 30 04:59:54.325129 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:54.325001    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf"
Oct 30 04:59:54.326101 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:54.326071    5898 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 04:59:54.326251 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:54.326143    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf" podUID="b5d063ac-6da8-4217-bddd-82a05db7bc50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 04:59:54.602939 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:54.602829    5898 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 04:59:54.603522 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:54.603488    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2" podUID="983a192c-67a4-4684-86fc-6df70f38a1ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 04:59:54.641694 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:54.641664    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:59:55.324484 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:55.324329    5898 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 04:59:55.324484 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:55.324392    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf" podUID="b5d063ac-6da8-4217-bddd-82a05db7bc50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 04:59:55.599992 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:55.599678    5898 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 04:59:55.599992 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:55.599736    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2" podUID="983a192c-67a4-4684-86fc-6df70f38a1ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 04:59:56.028376 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:56.028295    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 04:59:56.321978 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:56.321294    5898 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 04:59:56.321978 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:56.321361    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf" podUID="b5d063ac-6da8-4217-bddd-82a05db7bc50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 04:59:56.599838 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:56.599740    5898 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 04:59:56.600122 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:56.600081    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2" podUID="983a192c-67a4-4684-86fc-6df70f38a1ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 04:59:56.706450 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:56.706413    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:59:56.867023 ip-10-0-135-98 kubenswrapper[5898]: E1030 04:59:56.866943    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 04:59:57.328125 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:57.327302    5898 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 04:59:57.328125 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:57.327365    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf" podUID="b5d063ac-6da8-4217-bddd-82a05db7bc50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 04:59:57.596985 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:57.596343    5898 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 04:59:57.596985 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:57.596422    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2" podUID="983a192c-67a4-4684-86fc-6df70f38a1ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 04:59:58.322507 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:58.322458    5898 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 04:59:58.322842 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:58.322795    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf" podUID="b5d063ac-6da8-4217-bddd-82a05db7bc50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 04:59:58.593300 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:58.593209    5898 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 04:59:58.593830 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:58.593804    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2" podUID="983a192c-67a4-4684-86fc-6df70f38a1ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 04:59:59.200627 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:59.200356    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 04:59:59.336453 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:59.336418    5898 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 04:59:59.336741 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:59.336709    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf" podUID="b5d063ac-6da8-4217-bddd-82a05db7bc50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 04:59:59.593842 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:59.593805    5898 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 04:59:59.594386 ip-10-0-135-98 kubenswrapper[5898]: I1030 04:59:59.593858    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2" podUID="983a192c-67a4-4684-86fc-6df70f38a1ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 05:00:00.155680 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:00.155647    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888940-krx8r"]
Oct 30 05:00:00.161512 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:00.161472    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888940-zw8x6"]
Oct 30 05:00:00.161674 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:00.161612    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888940-krx8r"
Oct 30 05:00:00.168159 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:00.168129    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888940-zw8x6"
Oct 30 05:00:00.184963 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:00.184934    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888940-xwj6l"]
Oct 30 05:00:00.196417 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:00.196394    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888940-xwj6l"
Oct 30 05:00:00.225845 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:00.225816    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888940-krx8r"]
Oct 30 05:00:00.226777 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:00.226753    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888940-zw8x6"]
Oct 30 05:00:00.243509 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:00.243484    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888940-xwj6l"]
Oct 30 05:00:00.249505 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:00.249478    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/457ef20b-638c-4608-8b52-aa5f8d980596-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b21507-29888940-krx8r\" (UID: \"457ef20b-638c-4608-8b52-aa5f8d980596\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888940-krx8r"
Oct 30 05:00:00.249618 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:00.249594    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80719301-7cd9-4a45-9a68-5d9b604bab85-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-83e83e-29888940-xwj6l\" (UID: \"80719301-7cd9-4a45-9a68-5d9b604bab85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888940-xwj6l"
Oct 30 05:00:00.249666 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:00.249634    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfhh9\" (UniqueName: \"kubernetes.io/projected/457ef20b-638c-4608-8b52-aa5f8d980596-kube-api-access-vfhh9\") pod \"maas-api-key-cleanup-e2e-worker-w0-b21507-29888940-krx8r\" (UID: \"457ef20b-638c-4608-8b52-aa5f8d980596\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888940-krx8r"
Oct 30 05:00:00.249709 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:00.249673    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa13dfb9-89cc-46ed-b68a-5f99f8c50a56-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d55037-29888940-zw8x6\" (UID: \"aa13dfb9-89cc-46ed-b68a-5f99f8c50a56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888940-zw8x6"
Oct 30 05:00:00.249769 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:00.249754    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7csq\" (UniqueName: \"kubernetes.io/projected/aa13dfb9-89cc-46ed-b68a-5f99f8c50a56-kube-api-access-p7csq\") pod \"maas-api-key-cleanup-e2e-worker-w4-d55037-29888940-zw8x6\" (UID: \"aa13dfb9-89cc-46ed-b68a-5f99f8c50a56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888940-zw8x6"
Oct 30 05:00:00.249852 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:00.249840    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7879\" (UniqueName: \"kubernetes.io/projected/80719301-7cd9-4a45-9a68-5d9b604bab85-kube-api-access-p7879\") pod \"maas-api-key-cleanup-e2e-worker-w6-83e83e-29888940-xwj6l\" (UID: \"80719301-7cd9-4a45-9a68-5d9b604bab85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888940-xwj6l"
Oct 30 05:00:00.272635 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:00.272613    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\""
Oct 30 05:00:00.351146 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:00.351103    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7879\" (UniqueName: \"kubernetes.io/projected/80719301-7cd9-4a45-9a68-5d9b604bab85-kube-api-access-p7879\") pod \"maas-api-key-cleanup-e2e-worker-w6-83e83e-29888940-xwj6l\" (UID: \"80719301-7cd9-4a45-9a68-5d9b604bab85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888940-xwj6l"
Oct 30 05:00:00.351314 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:00.351245    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/457ef20b-638c-4608-8b52-aa5f8d980596-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b21507-29888940-krx8r\" (UID: \"457ef20b-638c-4608-8b52-aa5f8d980596\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888940-krx8r"
Oct 30 05:00:00.351314 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:00.351291    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80719301-7cd9-4a45-9a68-5d9b604bab85-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-83e83e-29888940-xwj6l\" (UID: \"80719301-7cd9-4a45-9a68-5d9b604bab85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888940-xwj6l"
Oct 30 05:00:00.351439 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:00.351312    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfhh9\" (UniqueName: \"kubernetes.io/projected/457ef20b-638c-4608-8b52-aa5f8d980596-kube-api-access-vfhh9\") pod \"maas-api-key-cleanup-e2e-worker-w0-b21507-29888940-krx8r\" (UID: \"457ef20b-638c-4608-8b52-aa5f8d980596\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888940-krx8r"
Oct 30 05:00:00.351439 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:00.351345    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa13dfb9-89cc-46ed-b68a-5f99f8c50a56-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d55037-29888940-zw8x6\" (UID: \"aa13dfb9-89cc-46ed-b68a-5f99f8c50a56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888940-zw8x6"
Oct 30 05:00:00.351439 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:00.351374    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7csq\" (UniqueName: \"kubernetes.io/projected/aa13dfb9-89cc-46ed-b68a-5f99f8c50a56-kube-api-access-p7csq\") pod \"maas-api-key-cleanup-e2e-worker-w4-d55037-29888940-zw8x6\" (UID: \"aa13dfb9-89cc-46ed-b68a-5f99f8c50a56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888940-zw8x6"
Oct 30 05:00:00.352157 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:00.352132    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/457ef20b-638c-4608-8b52-aa5f8d980596-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b21507-29888940-krx8r\" (UID: \"457ef20b-638c-4608-8b52-aa5f8d980596\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888940-krx8r"
Oct 30 05:00:00.352257 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:00.352174    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80719301-7cd9-4a45-9a68-5d9b604bab85-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-83e83e-29888940-xwj6l\" (UID: \"80719301-7cd9-4a45-9a68-5d9b604bab85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888940-xwj6l"
Oct 30 05:00:00.352322 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:00.352298    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa13dfb9-89cc-46ed-b68a-5f99f8c50a56-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d55037-29888940-zw8x6\" (UID: \"aa13dfb9-89cc-46ed-b68a-5f99f8c50a56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888940-zw8x6"
Oct 30 05:00:00.369403 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:00.369378    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7879\" (UniqueName: \"kubernetes.io/projected/80719301-7cd9-4a45-9a68-5d9b604bab85-kube-api-access-p7879\") pod \"maas-api-key-cleanup-e2e-worker-w6-83e83e-29888940-xwj6l\" (UID: \"80719301-7cd9-4a45-9a68-5d9b604bab85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888940-xwj6l"
Oct 30 05:00:00.369493 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:00.369465    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfhh9\" (UniqueName: \"kubernetes.io/projected/457ef20b-638c-4608-8b52-aa5f8d980596-kube-api-access-vfhh9\") pod \"maas-api-key-cleanup-e2e-worker-w0-b21507-29888940-krx8r\" (UID: \"457ef20b-638c-4608-8b52-aa5f8d980596\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888940-krx8r"
Oct 30 05:00:00.369679 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:00.369663    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7csq\" (UniqueName: \"kubernetes.io/projected/aa13dfb9-89cc-46ed-b68a-5f99f8c50a56-kube-api-access-p7csq\") pod \"maas-api-key-cleanup-e2e-worker-w4-d55037-29888940-zw8x6\" (UID: \"aa13dfb9-89cc-46ed-b68a-5f99f8c50a56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888940-zw8x6"
Oct 30 05:00:00.456541 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:00.456445    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf"
Oct 30 05:00:00.456973 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:00.456936    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf"
Oct 30 05:00:00.457790 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:00.457770    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf"
Oct 30 05:00:00.518893 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:00.518841    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888940-krx8r"
Oct 30 05:00:00.522806 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:00.522776    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888940-zw8x6"
Oct 30 05:00:00.541100 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:00.541064    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888940-xwj6l"
Oct 30 05:00:00.609066 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:00.605114    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2"
Oct 30 05:00:00.609066 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:00.607470    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2"
Oct 30 05:00:00.609808 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:00.609491    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2"
Oct 30 05:00:00.853659 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:00.853629    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888940-krx8r"]
Oct 30 05:00:00.855133 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:00:00.855107    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod457ef20b_638c_4608_8b52_aa5f8d980596.slice/crio-c62811b2c4d4636042f06150366b9a1fca7ab3dc77cfdb3bda62445142f22cf2 WatchSource:0}: Error finding container c62811b2c4d4636042f06150366b9a1fca7ab3dc77cfdb3bda62445142f22cf2: Status 404 returned error can't find the container with id c62811b2c4d4636042f06150366b9a1fca7ab3dc77cfdb3bda62445142f22cf2
Oct 30 05:00:00.873252 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:00.873230    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888940-zw8x6"]
Oct 30 05:00:00.875696 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:00:00.875667    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa13dfb9_89cc_46ed_b68a_5f99f8c50a56.slice/crio-0c4110473ce983db3791b1f31df82fdd7148d9edba3b55605feb1fc813705254 WatchSource:0}: Error finding container 0c4110473ce983db3791b1f31df82fdd7148d9edba3b55605feb1fc813705254: Status 404 returned error can't find the container with id 0c4110473ce983db3791b1f31df82fdd7148d9edba3b55605feb1fc813705254
Oct 30 05:00:00.892025 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:00.891999    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888940-xwj6l"]
Oct 30 05:00:00.893085 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:00:00.893057    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80719301_7cd9_4a45_9a68_5d9b604bab85.slice/crio-a4a9c9da8670b7195a3705067f09e5845cfdf97d46db846842d25edd0e2fab59 WatchSource:0}: Error finding container a4a9c9da8670b7195a3705067f09e5845cfdf97d46db846842d25edd0e2fab59: Status 404 returned error can't find the container with id a4a9c9da8670b7195a3705067f09e5845cfdf97d46db846842d25edd0e2fab59
Oct 30 05:00:01.153042 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:01.152949    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:00:01.427971 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:01.425327    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888940-zw8x6" event={"ID":"aa13dfb9-89cc-46ed-b68a-5f99f8c50a56","Type":"ContainerStarted","Data":"8dc7daabb6da2cdb37bffdb1f0b02a6243a2d962bd57afe12bf22b71f0e65673"}
Oct 30 05:00:01.427971 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:01.425379    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888940-zw8x6" event={"ID":"aa13dfb9-89cc-46ed-b68a-5f99f8c50a56","Type":"ContainerStarted","Data":"0c4110473ce983db3791b1f31df82fdd7148d9edba3b55605feb1fc813705254"}
Oct 30 05:00:01.427971 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:01.427548    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888940-xwj6l" event={"ID":"80719301-7cd9-4a45-9a68-5d9b604bab85","Type":"ContainerStarted","Data":"295f3f548a0d44640d56302778b7c25de58cd6b270e7555bbd90258ed17c95c6"}
Oct 30 05:00:01.427971 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:01.427596    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888940-xwj6l" event={"ID":"80719301-7cd9-4a45-9a68-5d9b604bab85","Type":"ContainerStarted","Data":"a4a9c9da8670b7195a3705067f09e5845cfdf97d46db846842d25edd0e2fab59"}
Oct 30 05:00:01.429335 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:01.429307    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888940-krx8r" event={"ID":"457ef20b-638c-4608-8b52-aa5f8d980596","Type":"ContainerStarted","Data":"52e3c42915fdc45e5ab5908f0c0c5c526a9930d407b88db2062b5aee194135e6"}
Oct 30 05:00:01.429490 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:01.429350    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888940-krx8r" event={"ID":"457ef20b-638c-4608-8b52-aa5f8d980596","Type":"ContainerStarted","Data":"c62811b2c4d4636042f06150366b9a1fca7ab3dc77cfdb3bda62445142f22cf2"}
Oct 30 05:00:01.500965 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:01.500896    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888940-krx8r" podStartSLOduration=1.500878593 podStartE2EDuration="1.500878593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:01.498123364 +0000 UTC m=+930.721943268" watchObservedRunningTime="2026-10-30 05:00:01.500878593 +0000 UTC m=+930.724698495"
Oct 30 05:00:01.501583 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:01.501555    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888940-zw8x6" podStartSLOduration=1.501546032 podStartE2EDuration="1.501546032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:01.461515158 +0000 UTC m=+930.685335059" watchObservedRunningTime="2026-10-30 05:00:01.501546032 +0000 UTC m=+930.725365933"
Oct 30 05:00:01.535744 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:01.535643    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888940-xwj6l" podStartSLOduration=1.535590561 podStartE2EDuration="1.535590561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:01.535304797 +0000 UTC m=+930.759124702" watchObservedRunningTime="2026-10-30 05:00:01.535590561 +0000 UTC m=+930.759410465"
Oct 30 05:00:02.437694 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:02.437658    5898 generic.go:356] "Generic (PLEG): container finished" podID="457ef20b-638c-4608-8b52-aa5f8d980596" containerID="52e3c42915fdc45e5ab5908f0c0c5c526a9930d407b88db2062b5aee194135e6" exitCode=60
Oct 30 05:00:02.438125 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:02.437739    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888940-krx8r" event={"ID":"457ef20b-638c-4608-8b52-aa5f8d980596","Type":"ContainerDied","Data":"52e3c42915fdc45e5ab5908f0c0c5c526a9930d407b88db2062b5aee194135e6"}
Oct 30 05:00:02.438125 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:02.437960    5898 scope.go:122] "RemoveContainer" containerID="52e3c42915fdc45e5ab5908f0c0c5c526a9930d407b88db2062b5aee194135e6"
Oct 30 05:00:02.439744 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:02.439718    5898 generic.go:356] "Generic (PLEG): container finished" podID="aa13dfb9-89cc-46ed-b68a-5f99f8c50a56" containerID="8dc7daabb6da2cdb37bffdb1f0b02a6243a2d962bd57afe12bf22b71f0e65673" exitCode=60
Oct 30 05:00:02.439911 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:02.439799    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888940-zw8x6" event={"ID":"aa13dfb9-89cc-46ed-b68a-5f99f8c50a56","Type":"ContainerDied","Data":"8dc7daabb6da2cdb37bffdb1f0b02a6243a2d962bd57afe12bf22b71f0e65673"}
Oct 30 05:00:02.440096 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:02.440081    5898 scope.go:122] "RemoveContainer" containerID="8dc7daabb6da2cdb37bffdb1f0b02a6243a2d962bd57afe12bf22b71f0e65673"
Oct 30 05:00:02.444333 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:02.444311    5898 generic.go:356] "Generic (PLEG): container finished" podID="80719301-7cd9-4a45-9a68-5d9b604bab85" containerID="295f3f548a0d44640d56302778b7c25de58cd6b270e7555bbd90258ed17c95c6" exitCode=60
Oct 30 05:00:02.444446 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:02.444397    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888940-xwj6l" event={"ID":"80719301-7cd9-4a45-9a68-5d9b604bab85","Type":"ContainerDied","Data":"295f3f548a0d44640d56302778b7c25de58cd6b270e7555bbd90258ed17c95c6"}
Oct 30 05:00:02.444639 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:02.444621    5898 scope.go:122] "RemoveContainer" containerID="295f3f548a0d44640d56302778b7c25de58cd6b270e7555bbd90258ed17c95c6"
Oct 30 05:00:03.008092 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:03.008013    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:00:03.451969 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:03.451935    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888940-zw8x6" event={"ID":"aa13dfb9-89cc-46ed-b68a-5f99f8c50a56","Type":"ContainerStarted","Data":"9abed01094bd6e7c216c6a01b4d2db53bc5b34ed318833e70f8e92e5095e2a59"}
Oct 30 05:00:03.453820 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:03.453798    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888940-xwj6l" event={"ID":"80719301-7cd9-4a45-9a68-5d9b604bab85","Type":"ContainerStarted","Data":"2b614531974873d54564afe0064bd1c2688dfa78e61d1bf806c4225000c75a5f"}
Oct 30 05:00:03.455693 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:03.455672    5898 generic.go:356] "Generic (PLEG): container finished" podID="457ef20b-638c-4608-8b52-aa5f8d980596" containerID="ddd7375a49c31265619cc90340e4e4aa085bf2af8d6c010e7f3d6a7e55aacb2f" exitCode=60
Oct 30 05:00:03.455794 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:03.455734    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888940-krx8r" event={"ID":"457ef20b-638c-4608-8b52-aa5f8d980596","Type":"ContainerDied","Data":"ddd7375a49c31265619cc90340e4e4aa085bf2af8d6c010e7f3d6a7e55aacb2f"}
Oct 30 05:00:03.455866 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:03.455785    5898 scope.go:122] "RemoveContainer" containerID="52e3c42915fdc45e5ab5908f0c0c5c526a9930d407b88db2062b5aee194135e6"
Oct 30 05:00:03.455976 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:03.455953    5898 scope.go:122] "RemoveContainer" containerID="ddd7375a49c31265619cc90340e4e4aa085bf2af8d6c010e7f3d6a7e55aacb2f"
Oct 30 05:00:03.456232 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:00:03.456214    5898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b21507-29888940-krx8r_odh-ai-gateway-infra(457ef20b-638c-4608-8b52-aa5f8d980596)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888940-krx8r" podUID="457ef20b-638c-4608-8b52-aa5f8d980596"
Oct 30 05:00:04.463624 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:04.463584    5898 generic.go:356] "Generic (PLEG): container finished" podID="aa13dfb9-89cc-46ed-b68a-5f99f8c50a56" containerID="9abed01094bd6e7c216c6a01b4d2db53bc5b34ed318833e70f8e92e5095e2a59" exitCode=60
Oct 30 05:00:04.464084 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:04.463664    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888940-zw8x6" event={"ID":"aa13dfb9-89cc-46ed-b68a-5f99f8c50a56","Type":"ContainerDied","Data":"9abed01094bd6e7c216c6a01b4d2db53bc5b34ed318833e70f8e92e5095e2a59"}
Oct 30 05:00:04.464084 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:04.463723    5898 scope.go:122] "RemoveContainer" containerID="8dc7daabb6da2cdb37bffdb1f0b02a6243a2d962bd57afe12bf22b71f0e65673"
Oct 30 05:00:04.464218 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:04.464201    5898 scope.go:122] "RemoveContainer" containerID="9abed01094bd6e7c216c6a01b4d2db53bc5b34ed318833e70f8e92e5095e2a59"
Oct 30 05:00:04.464543 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:00:04.464513    5898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d55037-29888940-zw8x6_odh-ai-gateway-infra(aa13dfb9-89cc-46ed-b68a-5f99f8c50a56)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888940-zw8x6" podUID="aa13dfb9-89cc-46ed-b68a-5f99f8c50a56"
Oct 30 05:00:04.465877 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:04.465857    5898 generic.go:356] "Generic (PLEG): container finished" podID="80719301-7cd9-4a45-9a68-5d9b604bab85" containerID="2b614531974873d54564afe0064bd1c2688dfa78e61d1bf806c4225000c75a5f" exitCode=60
Oct 30 05:00:04.465987 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:04.465944    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888940-xwj6l" event={"ID":"80719301-7cd9-4a45-9a68-5d9b604bab85","Type":"ContainerDied","Data":"2b614531974873d54564afe0064bd1c2688dfa78e61d1bf806c4225000c75a5f"}
Oct 30 05:00:04.466160 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:04.466146    5898 scope.go:122] "RemoveContainer" containerID="2b614531974873d54564afe0064bd1c2688dfa78e61d1bf806c4225000c75a5f"
Oct 30 05:00:04.466444 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:00:04.466427    5898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-83e83e-29888940-xwj6l_odh-ai-gateway-infra(80719301-7cd9-4a45-9a68-5d9b604bab85)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888940-xwj6l" podUID="80719301-7cd9-4a45-9a68-5d9b604bab85"
Oct 30 05:00:04.468158 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:04.468140    5898 scope.go:122] "RemoveContainer" containerID="ddd7375a49c31265619cc90340e4e4aa085bf2af8d6c010e7f3d6a7e55aacb2f"
Oct 30 05:00:04.468390 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:00:04.468373    5898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b21507-29888940-krx8r_odh-ai-gateway-infra(457ef20b-638c-4608-8b52-aa5f8d980596)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888940-krx8r" podUID="457ef20b-638c-4608-8b52-aa5f8d980596"
Oct 30 05:00:04.476300 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:04.476277    5898 scope.go:122] "RemoveContainer" containerID="295f3f548a0d44640d56302778b7c25de58cd6b270e7555bbd90258ed17c95c6"
Oct 30 05:00:04.687266 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:04.687232    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:00:05.475489 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:05.475457    5898 scope.go:122] "RemoveContainer" containerID="9abed01094bd6e7c216c6a01b4d2db53bc5b34ed318833e70f8e92e5095e2a59"
Oct 30 05:00:05.475991 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:00:05.475768    5898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d55037-29888940-zw8x6_odh-ai-gateway-infra(aa13dfb9-89cc-46ed-b68a-5f99f8c50a56)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888940-zw8x6" podUID="aa13dfb9-89cc-46ed-b68a-5f99f8c50a56"
Oct 30 05:00:05.477182 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:05.477155    5898 scope.go:122] "RemoveContainer" containerID="2b614531974873d54564afe0064bd1c2688dfa78e61d1bf806c4225000c75a5f"
Oct 30 05:00:05.477408 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:00:05.477390    5898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-83e83e-29888940-xwj6l_odh-ai-gateway-infra(80719301-7cd9-4a45-9a68-5d9b604bab85)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888940-xwj6l" podUID="80719301-7cd9-4a45-9a68-5d9b604bab85"
Oct 30 05:00:06.556549 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:06.556515    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:00:08.262122 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:08.262089    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:00:08.550154 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:08.550060    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g"]
Oct 30 05:00:08.555770 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:08.555746    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g"
Oct 30 05:00:08.571839 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:08.571808    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g"]
Oct 30 05:00:08.580326 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:08.580292    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-83e83e\"/\"distinct-2-w6-83e83e-kserve-self-signed-certs\""
Oct 30 05:00:08.592057 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:08.592024    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:00:08.650262 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:08.650229    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70294a85-1050-4632-b3bf-c3e44a4c0b41-home\") pod \"distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g\" (UID: \"70294a85-1050-4632-b3bf-c3e44a4c0b41\") " pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g"
Oct 30 05:00:08.650437 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:08.650271    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70294a85-1050-4632-b3bf-c3e44a4c0b41-model-cache\") pod \"distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g\" (UID: \"70294a85-1050-4632-b3bf-c3e44a4c0b41\") " pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g"
Oct 30 05:00:08.650510 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:08.650448    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70294a85-1050-4632-b3bf-c3e44a4c0b41-tmp-dir\") pod \"distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g\" (UID: \"70294a85-1050-4632-b3bf-c3e44a4c0b41\") " pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g"
Oct 30 05:00:08.650510 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:08.650501    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70294a85-1050-4632-b3bf-c3e44a4c0b41-tls-certs\") pod \"distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g\" (UID: \"70294a85-1050-4632-b3bf-c3e44a4c0b41\") " pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g"
Oct 30 05:00:08.650620 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:08.650589    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70294a85-1050-4632-b3bf-c3e44a4c0b41-dshm\") pod \"distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g\" (UID: \"70294a85-1050-4632-b3bf-c3e44a4c0b41\") " pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g"
Oct 30 05:00:08.650620 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:08.650608    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk2cm\" (UniqueName: \"kubernetes.io/projected/70294a85-1050-4632-b3bf-c3e44a4c0b41-kube-api-access-xk2cm\") pod \"distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g\" (UID: \"70294a85-1050-4632-b3bf-c3e44a4c0b41\") " pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g"
Oct 30 05:00:08.751947 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:08.751871    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70294a85-1050-4632-b3bf-c3e44a4c0b41-tmp-dir\") pod \"distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g\" (UID: \"70294a85-1050-4632-b3bf-c3e44a4c0b41\") " pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g"
Oct 30 05:00:08.752156 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:08.751979    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70294a85-1050-4632-b3bf-c3e44a4c0b41-tls-certs\") pod \"distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g\" (UID: \"70294a85-1050-4632-b3bf-c3e44a4c0b41\") " pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g"
Oct 30 05:00:08.752156 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:08.752024    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70294a85-1050-4632-b3bf-c3e44a4c0b41-dshm\") pod \"distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g\" (UID: \"70294a85-1050-4632-b3bf-c3e44a4c0b41\") " pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g"
Oct 30 05:00:08.752156 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:08.752049    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk2cm\" (UniqueName: \"kubernetes.io/projected/70294a85-1050-4632-b3bf-c3e44a4c0b41-kube-api-access-xk2cm\") pod \"distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g\" (UID: \"70294a85-1050-4632-b3bf-c3e44a4c0b41\") " pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g"
Oct 30 05:00:08.752156 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:08.752156    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70294a85-1050-4632-b3bf-c3e44a4c0b41-home\") pod \"distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g\" (UID: \"70294a85-1050-4632-b3bf-c3e44a4c0b41\") " pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g"
Oct 30 05:00:08.752370 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:08.752206    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70294a85-1050-4632-b3bf-c3e44a4c0b41-model-cache\") pod \"distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g\" (UID: \"70294a85-1050-4632-b3bf-c3e44a4c0b41\") " pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g"
Oct 30 05:00:08.752370 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:08.752330    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70294a85-1050-4632-b3bf-c3e44a4c0b41-tmp-dir\") pod \"distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g\" (UID: \"70294a85-1050-4632-b3bf-c3e44a4c0b41\") " pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g"
Oct 30 05:00:08.752653 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:08.752627    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70294a85-1050-4632-b3bf-c3e44a4c0b41-model-cache\") pod \"distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g\" (UID: \"70294a85-1050-4632-b3bf-c3e44a4c0b41\") " pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g"
Oct 30 05:00:08.753548 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:08.753197    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70294a85-1050-4632-b3bf-c3e44a4c0b41-home\") pod \"distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g\" (UID: \"70294a85-1050-4632-b3bf-c3e44a4c0b41\") " pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g"
Oct 30 05:00:08.756194 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:08.756172    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70294a85-1050-4632-b3bf-c3e44a4c0b41-dshm\") pod \"distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g\" (UID: \"70294a85-1050-4632-b3bf-c3e44a4c0b41\") " pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g"
Oct 30 05:00:08.757364 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:08.757322    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70294a85-1050-4632-b3bf-c3e44a4c0b41-tls-certs\") pod \"distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g\" (UID: \"70294a85-1050-4632-b3bf-c3e44a4c0b41\") " pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g"
Oct 30 05:00:08.779794 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:08.779760    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk2cm\" (UniqueName: \"kubernetes.io/projected/70294a85-1050-4632-b3bf-c3e44a4c0b41-kube-api-access-xk2cm\") pod \"distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g\" (UID: \"70294a85-1050-4632-b3bf-c3e44a4c0b41\") " pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g"
Oct 30 05:00:08.873158 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:08.873123    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g"
Oct 30 05:00:09.190112 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:09.190080    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g"]
Oct 30 05:00:09.191011 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:00:09.190983    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod70294a85_1050_4632_b3bf_c3e44a4c0b41.slice/crio-1cad023cb4608d77f4afa45b0ef20859c9171bb4d2139559ad1f8fa283b49245 WatchSource:0}: Error finding container 1cad023cb4608d77f4afa45b0ef20859c9171bb4d2139559ad1f8fa283b49245: Status 404 returned error can't find the container with id 1cad023cb4608d77f4afa45b0ef20859c9171bb4d2139559ad1f8fa283b49245
Oct 30 05:00:09.503345 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:09.503261    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g" event={"ID":"70294a85-1050-4632-b3bf-c3e44a4c0b41","Type":"ContainerStarted","Data":"2bae3ddc4b39eb73f211ad13b176ec58efec42ca6434a0166638f41b8b460e5e"}
Oct 30 05:00:09.503345 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:09.503309    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g" event={"ID":"70294a85-1050-4632-b3bf-c3e44a4c0b41","Type":"ContainerStarted","Data":"1cad023cb4608d77f4afa45b0ef20859c9171bb4d2139559ad1f8fa283b49245"}
Oct 30 05:00:09.541045 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:09.540988    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g" podStartSLOduration=1.540971103 podStartE2EDuration="1.540971103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:09.539669349 +0000 UTC m=+938.763489232" watchObservedRunningTime="2026-10-30 05:00:09.540971103 +0000 UTC m=+938.764791005"
Oct 30 05:00:09.761848 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:09.761765    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c862f32-f4ls6"]
Oct 30 05:00:09.765269 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:09.765249    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c862f32-f4ls6"
Oct 30 05:00:09.780419 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:09.780396    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:00:09.783775 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:09.783756    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c862f32-f4ls6"]
Oct 30 05:00:09.867964 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:09.867890    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73011279-ac38-4b5c-a830-6558bb34d850-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c862f32-f4ls6\" (UID: \"73011279-ac38-4b5c-a830-6558bb34d850\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c862f32-f4ls6"
Oct 30 05:00:09.868134 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:09.868018    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hn62\" (UniqueName: \"kubernetes.io/projected/73011279-ac38-4b5c-a830-6558bb34d850-kube-api-access-2hn62\") pod \"maas-api-revoke-sub-8c862f32-f4ls6\" (UID: \"73011279-ac38-4b5c-a830-6558bb34d850\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c862f32-f4ls6"
Oct 30 05:00:09.868191 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:09.868176    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73011279-ac38-4b5c-a830-6558bb34d850-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c862f32-f4ls6\" (UID: \"73011279-ac38-4b5c-a830-6558bb34d850\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c862f32-f4ls6"
Oct 30 05:00:09.874163 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:09.874138    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g"
Oct 30 05:00:09.969806 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:09.969772    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73011279-ac38-4b5c-a830-6558bb34d850-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c862f32-f4ls6\" (UID: \"73011279-ac38-4b5c-a830-6558bb34d850\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c862f32-f4ls6"
Oct 30 05:00:09.969806 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:09.969805    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hn62\" (UniqueName: \"kubernetes.io/projected/73011279-ac38-4b5c-a830-6558bb34d850-kube-api-access-2hn62\") pod \"maas-api-revoke-sub-8c862f32-f4ls6\" (UID: \"73011279-ac38-4b5c-a830-6558bb34d850\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c862f32-f4ls6"
Oct 30 05:00:09.970060 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:09.969887    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73011279-ac38-4b5c-a830-6558bb34d850-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c862f32-f4ls6\" (UID: \"73011279-ac38-4b5c-a830-6558bb34d850\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c862f32-f4ls6"
Oct 30 05:00:09.970605 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:09.970581    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73011279-ac38-4b5c-a830-6558bb34d850-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c862f32-f4ls6\" (UID: \"73011279-ac38-4b5c-a830-6558bb34d850\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c862f32-f4ls6"
Oct 30 05:00:09.972189 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:09.972163    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73011279-ac38-4b5c-a830-6558bb34d850-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c862f32-f4ls6\" (UID: \"73011279-ac38-4b5c-a830-6558bb34d850\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c862f32-f4ls6"
Oct 30 05:00:09.990074 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:09.990052    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hn62\" (UniqueName: \"kubernetes.io/projected/73011279-ac38-4b5c-a830-6558bb34d850-kube-api-access-2hn62\") pod \"maas-api-revoke-sub-8c862f32-f4ls6\" (UID: \"73011279-ac38-4b5c-a830-6558bb34d850\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c862f32-f4ls6"
Oct 30 05:00:10.079679 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:10.079653    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c862f32-f4ls6"
Oct 30 05:00:10.252267 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:10.252230    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:00:10.377650 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:10.377623    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c862f32-f4ls6"]
Oct 30 05:00:10.377821 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:00:10.377799    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73011279_ac38_4b5c_a830_6558bb34d850.slice/crio-cac874fc41d2f0fa36c567f3fdcd325f3724a692e73d670eaed81881c017b8c9 WatchSource:0}: Error finding container cac874fc41d2f0fa36c567f3fdcd325f3724a692e73d670eaed81881c017b8c9: Status 404 returned error can't find the container with id cac874fc41d2f0fa36c567f3fdcd325f3724a692e73d670eaed81881c017b8c9
Oct 30 05:00:10.510533 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:10.510497    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c862f32-f4ls6" event={"ID":"73011279-ac38-4b5c-a830-6558bb34d850","Type":"ContainerStarted","Data":"cac874fc41d2f0fa36c567f3fdcd325f3724a692e73d670eaed81881c017b8c9"}
Oct 30 05:00:11.518726 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:11.518689    5898 generic.go:356] "Generic (PLEG): container finished" podID="73011279-ac38-4b5c-a830-6558bb34d850" containerID="d986aaa1c60a4e8b83b62a97ac81d10dbe97c96b9bc26b4574d3687156115840" exitCode=0
Oct 30 05:00:11.519188 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:11.518720    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c862f32-f4ls6" event={"ID":"73011279-ac38-4b5c-a830-6558bb34d850","Type":"ContainerDied","Data":"d986aaa1c60a4e8b83b62a97ac81d10dbe97c96b9bc26b4574d3687156115840"}
Oct 30 05:00:12.013388 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:12.013355    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:00:12.114154 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:12.114120    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:00:12.669644 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:12.669615    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c862f32-f4ls6"
Oct 30 05:00:12.806623 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:12.806537    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73011279-ac38-4b5c-a830-6558bb34d850-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73011279-ac38-4b5c-a830-6558bb34d850-maas-api-xks-ca\") pod \"73011279-ac38-4b5c-a830-6558bb34d850\" (UID: \"73011279-ac38-4b5c-a830-6558bb34d850\") "
Oct 30 05:00:12.806790 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:12.806638    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73011279-ac38-4b5c-a830-6558bb34d850-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73011279-ac38-4b5c-a830-6558bb34d850-maas-api-openshift-ca\") pod \"73011279-ac38-4b5c-a830-6558bb34d850\" (UID: \"73011279-ac38-4b5c-a830-6558bb34d850\") "
Oct 30 05:00:12.806790 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:12.806674    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73011279-ac38-4b5c-a830-6558bb34d850-kube-api-access-2hn62\" (UniqueName: \"kubernetes.io/projected/73011279-ac38-4b5c-a830-6558bb34d850-kube-api-access-2hn62\") pod \"73011279-ac38-4b5c-a830-6558bb34d850\" (UID: \"73011279-ac38-4b5c-a830-6558bb34d850\") "
Oct 30 05:00:12.807020 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:12.806982    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73011279-ac38-4b5c-a830-6558bb34d850-maas-api-openshift-ca" pod "73011279-ac38-4b5c-a830-6558bb34d850" (UID: "73011279-ac38-4b5c-a830-6558bb34d850"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:00:12.807232 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:12.807211    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73011279-ac38-4b5c-a830-6558bb34d850-maas-api-openshift-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:00:12.808706 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:12.808681    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73011279-ac38-4b5c-a830-6558bb34d850-maas-api-xks-ca" pod "73011279-ac38-4b5c-a830-6558bb34d850" (UID: "73011279-ac38-4b5c-a830-6558bb34d850"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:00:12.808782 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:12.808746    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73011279-ac38-4b5c-a830-6558bb34d850-kube-api-access-2hn62" pod "73011279-ac38-4b5c-a830-6558bb34d850" (UID: "73011279-ac38-4b5c-a830-6558bb34d850"). InnerVolumeSpecName "kube-api-access-2hn62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:00:12.908752 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:12.908728    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73011279-ac38-4b5c-a830-6558bb34d850-maas-api-xks-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:00:12.908752 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:12.908753    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hn62\" (UniqueName: \"kubernetes.io/projected/73011279-ac38-4b5c-a830-6558bb34d850-kube-api-access-2hn62\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:00:13.532943 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:13.532895    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c862f32-f4ls6"
Oct 30 05:00:13.533122 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:13.532995    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c862f32-f4ls6" event={"ID":"73011279-ac38-4b5c-a830-6558bb34d850","Type":"ContainerDied","Data":"cac874fc41d2f0fa36c567f3fdcd325f3724a692e73d670eaed81881c017b8c9"}
Oct 30 05:00:13.533122 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:13.533043    5898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cac874fc41d2f0fa36c567f3fdcd325f3724a692e73d670eaed81881c017b8c9"
Oct 30 05:00:14.056148 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:14.056115    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:00:14.889914 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:14.889882    5898 scope.go:122] "RemoveContainer" containerID="2b614531974873d54564afe0064bd1c2688dfa78e61d1bf806c4225000c75a5f"
Oct 30 05:00:14.890121 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:14.889983    5898 scope.go:122] "RemoveContainer" containerID="ddd7375a49c31265619cc90340e4e4aa085bf2af8d6c010e7f3d6a7e55aacb2f"
Oct 30 05:00:15.546113 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:15.546016    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888940-xwj6l" event={"ID":"80719301-7cd9-4a45-9a68-5d9b604bab85","Type":"ContainerStarted","Data":"ad3a69dff776473f1ea0ff02536301a04ed10d992f814d7116dec8b1f9ce97e7"}
Oct 30 05:00:15.547939 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:15.547893    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888940-krx8r" event={"ID":"457ef20b-638c-4608-8b52-aa5f8d980596","Type":"ContainerStarted","Data":"165f2b10ce0105207182263bbe03e9da2ad651e6496cab09b0b321e0773feb87"}
Oct 30 05:00:15.783054 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:15.783017    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:00:15.884181 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:15.884145    5898 scope.go:122] "RemoveContainer" containerID="9abed01094bd6e7c216c6a01b4d2db53bc5b34ed318833e70f8e92e5095e2a59"
Oct 30 05:00:16.556851 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:16.556763    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888940-zw8x6" event={"ID":"aa13dfb9-89cc-46ed-b68a-5f99f8c50a56","Type":"ContainerStarted","Data":"2d91ca39ccbaaa87cd506bcb5c92dfa378ede57a7c50862ff277d9c83ed6965f"}
Oct 30 05:00:16.558709 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:16.558682    5898 generic.go:356] "Generic (PLEG): container finished" podID="80719301-7cd9-4a45-9a68-5d9b604bab85" containerID="ad3a69dff776473f1ea0ff02536301a04ed10d992f814d7116dec8b1f9ce97e7" exitCode=60
Oct 30 05:00:16.558838 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:16.558746    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888940-xwj6l" event={"ID":"80719301-7cd9-4a45-9a68-5d9b604bab85","Type":"ContainerDied","Data":"ad3a69dff776473f1ea0ff02536301a04ed10d992f814d7116dec8b1f9ce97e7"}
Oct 30 05:00:16.558838 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:16.558795    5898 scope.go:122] "RemoveContainer" containerID="2b614531974873d54564afe0064bd1c2688dfa78e61d1bf806c4225000c75a5f"
Oct 30 05:00:16.559147 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:16.559129    5898 scope.go:122] "RemoveContainer" containerID="ad3a69dff776473f1ea0ff02536301a04ed10d992f814d7116dec8b1f9ce97e7"
Oct 30 05:00:16.559410 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:00:16.559384    5898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-83e83e-29888940-xwj6l_odh-ai-gateway-infra(80719301-7cd9-4a45-9a68-5d9b604bab85)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888940-xwj6l" podUID="80719301-7cd9-4a45-9a68-5d9b604bab85"
Oct 30 05:00:16.560727 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:16.560708    5898 generic.go:356] "Generic (PLEG): container finished" podID="457ef20b-638c-4608-8b52-aa5f8d980596" containerID="165f2b10ce0105207182263bbe03e9da2ad651e6496cab09b0b321e0773feb87" exitCode=60
Oct 30 05:00:16.560795 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:16.560736    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888940-krx8r" event={"ID":"457ef20b-638c-4608-8b52-aa5f8d980596","Type":"ContainerDied","Data":"165f2b10ce0105207182263bbe03e9da2ad651e6496cab09b0b321e0773feb87"}
Oct 30 05:00:16.561007 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:16.560994    5898 scope.go:122] "RemoveContainer" containerID="165f2b10ce0105207182263bbe03e9da2ad651e6496cab09b0b321e0773feb87"
Oct 30 05:00:16.561192 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:00:16.561176    5898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b21507-29888940-krx8r_odh-ai-gateway-infra(457ef20b-638c-4608-8b52-aa5f8d980596)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888940-krx8r" podUID="457ef20b-638c-4608-8b52-aa5f8d980596"
Oct 30 05:00:16.575073 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:16.575051    5898 scope.go:122] "RemoveContainer" containerID="ddd7375a49c31265619cc90340e4e4aa085bf2af8d6c010e7f3d6a7e55aacb2f"
Oct 30 05:00:16.603657 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:16.603629    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888940-xwj6l"]
Oct 30 05:00:16.635941 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:16.635911    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888940-krx8r"]
Oct 30 05:00:17.545589 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:17.545558    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:00:17.571628 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:17.571590    5898 generic.go:356] "Generic (PLEG): container finished" podID="aa13dfb9-89cc-46ed-b68a-5f99f8c50a56" containerID="2d91ca39ccbaaa87cd506bcb5c92dfa378ede57a7c50862ff277d9c83ed6965f" exitCode=60
Oct 30 05:00:17.572145 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:17.571710    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888940-zw8x6" event={"ID":"aa13dfb9-89cc-46ed-b68a-5f99f8c50a56","Type":"ContainerDied","Data":"2d91ca39ccbaaa87cd506bcb5c92dfa378ede57a7c50862ff277d9c83ed6965f"}
Oct 30 05:00:17.572145 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:17.571823    5898 scope.go:122] "RemoveContainer" containerID="9abed01094bd6e7c216c6a01b4d2db53bc5b34ed318833e70f8e92e5095e2a59"
Oct 30 05:00:17.572694 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:17.572381    5898 scope.go:122] "RemoveContainer" containerID="2d91ca39ccbaaa87cd506bcb5c92dfa378ede57a7c50862ff277d9c83ed6965f"
Oct 30 05:00:17.572930 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:00:17.572882    5898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d55037-29888940-zw8x6_odh-ai-gateway-infra(aa13dfb9-89cc-46ed-b68a-5f99f8c50a56)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888940-zw8x6" podUID="aa13dfb9-89cc-46ed-b68a-5f99f8c50a56"
Oct 30 05:00:17.623860 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:17.618325    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888940-zw8x6"]
Oct 30 05:00:17.802855 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:17.802799    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888940-krx8r"
Oct 30 05:00:17.806049 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:17.806030    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888940-xwj6l"
Oct 30 05:00:17.864049 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:17.864017    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80719301-7cd9-4a45-9a68-5d9b604bab85-kube-api-access-p7879\" (UniqueName: \"kubernetes.io/projected/80719301-7cd9-4a45-9a68-5d9b604bab85-kube-api-access-p7879\") pod \"80719301-7cd9-4a45-9a68-5d9b604bab85\" (UID: \"80719301-7cd9-4a45-9a68-5d9b604bab85\") "
Oct 30 05:00:17.864212 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:17.864072    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80719301-7cd9-4a45-9a68-5d9b604bab85-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80719301-7cd9-4a45-9a68-5d9b604bab85-maas-api-service-ca\") pod \"80719301-7cd9-4a45-9a68-5d9b604bab85\" (UID: \"80719301-7cd9-4a45-9a68-5d9b604bab85\") "
Oct 30 05:00:17.864212 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:17.864111    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/457ef20b-638c-4608-8b52-aa5f8d980596-kube-api-access-vfhh9\" (UniqueName: \"kubernetes.io/projected/457ef20b-638c-4608-8b52-aa5f8d980596-kube-api-access-vfhh9\") pod \"457ef20b-638c-4608-8b52-aa5f8d980596\" (UID: \"457ef20b-638c-4608-8b52-aa5f8d980596\") "
Oct 30 05:00:17.864212 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:17.864138    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/457ef20b-638c-4608-8b52-aa5f8d980596-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/457ef20b-638c-4608-8b52-aa5f8d980596-maas-api-service-ca\") pod \"457ef20b-638c-4608-8b52-aa5f8d980596\" (UID: \"457ef20b-638c-4608-8b52-aa5f8d980596\") "
Oct 30 05:00:17.864556 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:17.864524    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80719301-7cd9-4a45-9a68-5d9b604bab85-maas-api-service-ca" pod "80719301-7cd9-4a45-9a68-5d9b604bab85" (UID: "80719301-7cd9-4a45-9a68-5d9b604bab85"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:00:17.864556 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:17.864536    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/457ef20b-638c-4608-8b52-aa5f8d980596-maas-api-service-ca" pod "457ef20b-638c-4608-8b52-aa5f8d980596" (UID: "457ef20b-638c-4608-8b52-aa5f8d980596"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:00:17.864739 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:17.864685    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80719301-7cd9-4a45-9a68-5d9b604bab85-maas-api-service-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:00:17.864739 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:17.864708    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/457ef20b-638c-4608-8b52-aa5f8d980596-maas-api-service-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:00:17.866375 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:17.866341    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/457ef20b-638c-4608-8b52-aa5f8d980596-kube-api-access-vfhh9" pod "457ef20b-638c-4608-8b52-aa5f8d980596" (UID: "457ef20b-638c-4608-8b52-aa5f8d980596"). InnerVolumeSpecName "kube-api-access-vfhh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:00:17.866720 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:17.866696    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80719301-7cd9-4a45-9a68-5d9b604bab85-kube-api-access-p7879" pod "80719301-7cd9-4a45-9a68-5d9b604bab85" (UID: "80719301-7cd9-4a45-9a68-5d9b604bab85"). InnerVolumeSpecName "kube-api-access-p7879". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:00:17.965574 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:17.965541    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7879\" (UniqueName: \"kubernetes.io/projected/80719301-7cd9-4a45-9a68-5d9b604bab85-kube-api-access-p7879\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:00:17.965574 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:17.965569    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfhh9\" (UniqueName: \"kubernetes.io/projected/457ef20b-638c-4608-8b52-aa5f8d980596-kube-api-access-vfhh9\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:00:18.442445 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:18.442412    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e92d9e6-ctg5b"]
Oct 30 05:00:18.442975 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:18.442958    5898 cpu_manager.go:414] "containerMap: removing container" podUID="80719301-7cd9-4a45-9a68-5d9b604bab85" containerName="cleanup"
Oct 30 05:00:18.443082 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:18.442977    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80719301-7cd9-4a45-9a68-5d9b604bab85" containerName="cleanup"
Oct 30 05:00:18.443082 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:18.442992    5898 cpu_manager.go:414] "containerMap: removing container" podUID="457ef20b-638c-4608-8b52-aa5f8d980596" containerName="cleanup"
Oct 30 05:00:18.443082 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:18.442999    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="457ef20b-638c-4608-8b52-aa5f8d980596" containerName="cleanup"
Oct 30 05:00:18.443082 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:18.443020    5898 cpu_manager.go:414] "containerMap: removing container" podUID="73011279-ac38-4b5c-a830-6558bb34d850" containerName="revoke-keys"
Oct 30 05:00:18.443082 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:18.443028    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73011279-ac38-4b5c-a830-6558bb34d850" containerName="revoke-keys"
Oct 30 05:00:18.443082 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:18.443052    5898 cpu_manager.go:414] "containerMap: removing container" podUID="80719301-7cd9-4a45-9a68-5d9b604bab85" containerName="cleanup"
Oct 30 05:00:18.443082 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:18.443059    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80719301-7cd9-4a45-9a68-5d9b604bab85" containerName="cleanup"
Oct 30 05:00:18.443082 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:18.443070    5898 cpu_manager.go:414] "containerMap: removing container" podUID="457ef20b-638c-4608-8b52-aa5f8d980596" containerName="cleanup"
Oct 30 05:00:18.443082 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:18.443076    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="457ef20b-638c-4608-8b52-aa5f8d980596" containerName="cleanup"
Oct 30 05:00:18.443502 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:18.443232    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="457ef20b-638c-4608-8b52-aa5f8d980596" containerName="cleanup"
Oct 30 05:00:18.443502 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:18.443249    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="80719301-7cd9-4a45-9a68-5d9b604bab85" containerName="cleanup"
Oct 30 05:00:18.443502 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:18.443258    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="80719301-7cd9-4a45-9a68-5d9b604bab85" containerName="cleanup"
Oct 30 05:00:18.443502 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:18.443266    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="457ef20b-638c-4608-8b52-aa5f8d980596" containerName="cleanup"
Oct 30 05:00:18.443502 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:18.443276    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="73011279-ac38-4b5c-a830-6558bb34d850" containerName="revoke-keys"
Oct 30 05:00:18.443502 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:18.443287    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="80719301-7cd9-4a45-9a68-5d9b604bab85" containerName="cleanup"
Oct 30 05:00:18.443502 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:18.443297    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="457ef20b-638c-4608-8b52-aa5f8d980596" containerName="cleanup"
Oct 30 05:00:18.446666 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:18.446646    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e92d9e6-ctg5b"
Oct 30 05:00:18.464049 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:18.464024    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e92d9e6-ctg5b"]
Oct 30 05:00:18.472135 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:18.472112    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:00:18.472247 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:18.472217    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82219032-2bf7-4bc8-89d3-5063a7ab1f0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e92d9e6-ctg5b\" (UID: \"82219032-2bf7-4bc8-89d3-5063a7ab1f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e92d9e6-ctg5b"
Oct 30 05:00:18.472326 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:18.472260    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqb5s\" (UniqueName: \"kubernetes.io/projected/82219032-2bf7-4bc8-89d3-5063a7ab1f0f-kube-api-access-bqb5s\") pod \"maas-api-revoke-sub-3e92d9e6-ctg5b\" (UID: \"82219032-2bf7-4bc8-89d3-5063a7ab1f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e92d9e6-ctg5b"
Oct 30 05:00:18.472326 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:18.472316    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82219032-2bf7-4bc8-89d3-5063a7ab1f0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e92d9e6-ctg5b\" (UID: \"82219032-2bf7-4bc8-89d3-5063a7ab1f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e92d9e6-ctg5b"
Oct 30 05:00:18.573837 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:18.573798    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82219032-2bf7-4bc8-89d3-5063a7ab1f0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e92d9e6-ctg5b\" (UID: \"82219032-2bf7-4bc8-89d3-5063a7ab1f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e92d9e6-ctg5b"
Oct 30 05:00:18.574282 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:18.573862    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqb5s\" (UniqueName: \"kubernetes.io/projected/82219032-2bf7-4bc8-89d3-5063a7ab1f0f-kube-api-access-bqb5s\") pod \"maas-api-revoke-sub-3e92d9e6-ctg5b\" (UID: \"82219032-2bf7-4bc8-89d3-5063a7ab1f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e92d9e6-ctg5b"
Oct 30 05:00:18.574282 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:18.573951    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82219032-2bf7-4bc8-89d3-5063a7ab1f0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e92d9e6-ctg5b\" (UID: \"82219032-2bf7-4bc8-89d3-5063a7ab1f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e92d9e6-ctg5b"
Oct 30 05:00:18.574414 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:18.574394    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82219032-2bf7-4bc8-89d3-5063a7ab1f0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e92d9e6-ctg5b\" (UID: \"82219032-2bf7-4bc8-89d3-5063a7ab1f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e92d9e6-ctg5b"
Oct 30 05:00:18.576312 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:18.576288    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82219032-2bf7-4bc8-89d3-5063a7ab1f0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e92d9e6-ctg5b\" (UID: \"82219032-2bf7-4bc8-89d3-5063a7ab1f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e92d9e6-ctg5b"
Oct 30 05:00:18.588332 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:18.588291    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888940-xwj6l"
Oct 30 05:00:18.588614 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:18.588288    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888940-xwj6l" event={"ID":"80719301-7cd9-4a45-9a68-5d9b604bab85","Type":"ContainerDied","Data":"a4a9c9da8670b7195a3705067f09e5845cfdf97d46db846842d25edd0e2fab59"}
Oct 30 05:00:18.588733 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:18.588624    5898 scope.go:122] "RemoveContainer" containerID="ad3a69dff776473f1ea0ff02536301a04ed10d992f814d7116dec8b1f9ce97e7"
Oct 30 05:00:18.590055 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:18.590034    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888940-krx8r"
Oct 30 05:00:18.590174 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:18.590065    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888940-krx8r" event={"ID":"457ef20b-638c-4608-8b52-aa5f8d980596","Type":"ContainerDied","Data":"c62811b2c4d4636042f06150366b9a1fca7ab3dc77cfdb3bda62445142f22cf2"}
Oct 30 05:00:18.596747 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:18.596727    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqb5s\" (UniqueName: \"kubernetes.io/projected/82219032-2bf7-4bc8-89d3-5063a7ab1f0f-kube-api-access-bqb5s\") pod \"maas-api-revoke-sub-3e92d9e6-ctg5b\" (UID: \"82219032-2bf7-4bc8-89d3-5063a7ab1f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e92d9e6-ctg5b"
Oct 30 05:00:18.600182 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:18.600159    5898 scope.go:122] "RemoveContainer" containerID="165f2b10ce0105207182263bbe03e9da2ad651e6496cab09b0b321e0773feb87"
Oct 30 05:00:18.659795 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:18.659765    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888940-krx8r"]
Oct 30 05:00:18.665967 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:18.665939    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888940-krx8r"]
Oct 30 05:00:18.705846 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:18.705775    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888940-xwj6l"]
Oct 30 05:00:18.713384 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:18.713354    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888940-xwj6l"]
Oct 30 05:00:18.728317 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:18.728294    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888940-zw8x6"
Oct 30 05:00:18.760207 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:18.760185    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e92d9e6-ctg5b"
Oct 30 05:00:18.776112 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:18.776086    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa13dfb9-89cc-46ed-b68a-5f99f8c50a56-kube-api-access-p7csq\" (UniqueName: \"kubernetes.io/projected/aa13dfb9-89cc-46ed-b68a-5f99f8c50a56-kube-api-access-p7csq\") pod \"aa13dfb9-89cc-46ed-b68a-5f99f8c50a56\" (UID: \"aa13dfb9-89cc-46ed-b68a-5f99f8c50a56\") "
Oct 30 05:00:18.776252 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:18.776173    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa13dfb9-89cc-46ed-b68a-5f99f8c50a56-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa13dfb9-89cc-46ed-b68a-5f99f8c50a56-maas-api-service-ca\") pod \"aa13dfb9-89cc-46ed-b68a-5f99f8c50a56\" (UID: \"aa13dfb9-89cc-46ed-b68a-5f99f8c50a56\") "
Oct 30 05:00:18.776878 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:18.776707    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa13dfb9-89cc-46ed-b68a-5f99f8c50a56-maas-api-service-ca" pod "aa13dfb9-89cc-46ed-b68a-5f99f8c50a56" (UID: "aa13dfb9-89cc-46ed-b68a-5f99f8c50a56"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:00:18.776878 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:18.776846    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa13dfb9-89cc-46ed-b68a-5f99f8c50a56-maas-api-service-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:00:18.778331 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:18.778310    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa13dfb9-89cc-46ed-b68a-5f99f8c50a56-kube-api-access-p7csq" pod "aa13dfb9-89cc-46ed-b68a-5f99f8c50a56" (UID: "aa13dfb9-89cc-46ed-b68a-5f99f8c50a56"). InnerVolumeSpecName "kube-api-access-p7csq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:00:18.873515 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:18.873486    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g"
Oct 30 05:00:18.877482 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:18.877456    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7csq\" (UniqueName: \"kubernetes.io/projected/aa13dfb9-89cc-46ed-b68a-5f99f8c50a56-kube-api-access-p7csq\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:00:18.893583 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:18.893549    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="457ef20b-638c-4608-8b52-aa5f8d980596" path="/var/lib/kubelet/pods/457ef20b-638c-4608-8b52-aa5f8d980596/volumes"
Oct 30 05:00:18.894224 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:18.894197    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80719301-7cd9-4a45-9a68-5d9b604bab85" path="/var/lib/kubelet/pods/80719301-7cd9-4a45-9a68-5d9b604bab85/volumes"
Oct 30 05:00:18.894886 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:18.894861    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g"
Oct 30 05:00:19.068016 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:19.067985    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e92d9e6-ctg5b"]
Oct 30 05:00:19.068445 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:00:19.068417    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82219032_2bf7_4bc8_89d3_5063a7ab1f0f.slice/crio-f9dda853127dbbad16518da4a25a3e4bf155efb8206407ae02283ac28e7cf60e WatchSource:0}: Error finding container f9dda853127dbbad16518da4a25a3e4bf155efb8206407ae02283ac28e7cf60e: Status 404 returned error can't find the container with id f9dda853127dbbad16518da4a25a3e4bf155efb8206407ae02283ac28e7cf60e
Oct 30 05:00:19.373700 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:19.373671    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:00:19.604736 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:19.604681    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e92d9e6-ctg5b" event={"ID":"82219032-2bf7-4bc8-89d3-5063a7ab1f0f","Type":"ContainerStarted","Data":"1b7e6b22c423f1001c5dde6b568377210f1501c670537b2484c29dc90102a0b0"}
Oct 30 05:00:19.605202 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:19.604786    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e92d9e6-ctg5b" event={"ID":"82219032-2bf7-4bc8-89d3-5063a7ab1f0f","Type":"ContainerStarted","Data":"f9dda853127dbbad16518da4a25a3e4bf155efb8206407ae02283ac28e7cf60e"}
Oct 30 05:00:19.606184 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:19.606157    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888940-zw8x6" event={"ID":"aa13dfb9-89cc-46ed-b68a-5f99f8c50a56","Type":"ContainerDied","Data":"0c4110473ce983db3791b1f31df82fdd7148d9edba3b55605feb1fc813705254"}
Oct 30 05:00:19.606299 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:19.606218    5898 scope.go:122] "RemoveContainer" containerID="2d91ca39ccbaaa87cd506bcb5c92dfa378ede57a7c50862ff277d9c83ed6965f"
Oct 30 05:00:19.606299 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:19.606168    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888940-zw8x6"
Oct 30 05:00:19.624922 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:19.624855    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g"
Oct 30 05:00:19.652355 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:19.652312    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e92d9e6-ctg5b" podStartSLOduration=1.652299366 podStartE2EDuration="1.652299366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:19.650173092 +0000 UTC m=+948.873992993" watchObservedRunningTime="2026-10-30 05:00:19.652299366 +0000 UTC m=+948.876119266"
Oct 30 05:00:19.689993 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:19.689965    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888940-zw8x6"]
Oct 30 05:00:19.695352 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:19.695324    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d55037-29888940-zw8x6"]
Oct 30 05:00:20.617266 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:20.617233    5898 generic.go:356] "Generic (PLEG): container finished" podID="82219032-2bf7-4bc8-89d3-5063a7ab1f0f" containerID="1b7e6b22c423f1001c5dde6b568377210f1501c670537b2484c29dc90102a0b0" exitCode=0
Oct 30 05:00:20.617715 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:20.617319    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e92d9e6-ctg5b" event={"ID":"82219032-2bf7-4bc8-89d3-5063a7ab1f0f","Type":"ContainerDied","Data":"1b7e6b22c423f1001c5dde6b568377210f1501c670537b2484c29dc90102a0b0"}
Oct 30 05:00:20.891343 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:20.891260    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa13dfb9-89cc-46ed-b68a-5f99f8c50a56" path="/var/lib/kubelet/pods/aa13dfb9-89cc-46ed-b68a-5f99f8c50a56/volumes"
Oct 30 05:00:21.045823 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.045792    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:00:21.702464 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.702436    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft"]
Oct 30 05:00:21.702992 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.702974    5898 cpu_manager.go:414] "containerMap: removing container" podUID="457ef20b-638c-4608-8b52-aa5f8d980596" containerName="cleanup"
Oct 30 05:00:21.702992 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.702991    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="457ef20b-638c-4608-8b52-aa5f8d980596" containerName="cleanup"
Oct 30 05:00:21.703147 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.703014    5898 cpu_manager.go:414] "containerMap: removing container" podUID="aa13dfb9-89cc-46ed-b68a-5f99f8c50a56" containerName="cleanup"
Oct 30 05:00:21.703147 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.703022    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa13dfb9-89cc-46ed-b68a-5f99f8c50a56" containerName="cleanup"
Oct 30 05:00:21.703147 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.703033    5898 cpu_manager.go:414] "containerMap: removing container" podUID="aa13dfb9-89cc-46ed-b68a-5f99f8c50a56" containerName="cleanup"
Oct 30 05:00:21.703147 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.703041    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa13dfb9-89cc-46ed-b68a-5f99f8c50a56" containerName="cleanup"
Oct 30 05:00:21.703147 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.703082    5898 cpu_manager.go:414] "containerMap: removing container" podUID="80719301-7cd9-4a45-9a68-5d9b604bab85" containerName="cleanup"
Oct 30 05:00:21.703147 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.703090    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80719301-7cd9-4a45-9a68-5d9b604bab85" containerName="cleanup"
Oct 30 05:00:21.703388 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.703226    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa13dfb9-89cc-46ed-b68a-5f99f8c50a56" containerName="cleanup"
Oct 30 05:00:21.703388 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.703239    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa13dfb9-89cc-46ed-b68a-5f99f8c50a56" containerName="cleanup"
Oct 30 05:00:21.706894 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.706874    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft"
Oct 30 05:00:21.733940 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.731836    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft"]
Oct 30 05:00:21.739954 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.739894    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq"]
Oct 30 05:00:21.740504 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.740483    5898 cpu_manager.go:414] "containerMap: removing container" podUID="aa13dfb9-89cc-46ed-b68a-5f99f8c50a56" containerName="cleanup"
Oct 30 05:00:21.740504 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.740504    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa13dfb9-89cc-46ed-b68a-5f99f8c50a56" containerName="cleanup"
Oct 30 05:00:21.741105 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.741083    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa13dfb9-89cc-46ed-b68a-5f99f8c50a56" containerName="cleanup"
Oct 30 05:00:21.744395 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.744376    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq"
Oct 30 05:00:21.764138 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.764110    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq"]
Oct 30 05:00:21.794084 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.794061    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e92d9e6-ctg5b"
Oct 30 05:00:21.808422 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.808395    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82219032-2bf7-4bc8-89d3-5063a7ab1f0f-kube-api-access-bqb5s\" (UniqueName: \"kubernetes.io/projected/82219032-2bf7-4bc8-89d3-5063a7ab1f0f-kube-api-access-bqb5s\") pod \"82219032-2bf7-4bc8-89d3-5063a7ab1f0f\" (UID: \"82219032-2bf7-4bc8-89d3-5063a7ab1f0f\") "
Oct 30 05:00:21.808562 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.808498    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82219032-2bf7-4bc8-89d3-5063a7ab1f0f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82219032-2bf7-4bc8-89d3-5063a7ab1f0f-maas-api-xks-ca\") pod \"82219032-2bf7-4bc8-89d3-5063a7ab1f0f\" (UID: \"82219032-2bf7-4bc8-89d3-5063a7ab1f0f\") "
Oct 30 05:00:21.808562 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.808534    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82219032-2bf7-4bc8-89d3-5063a7ab1f0f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82219032-2bf7-4bc8-89d3-5063a7ab1f0f-maas-api-openshift-ca\") pod \"82219032-2bf7-4bc8-89d3-5063a7ab1f0f\" (UID: \"82219032-2bf7-4bc8-89d3-5063a7ab1f0f\") "
Oct 30 05:00:21.808889 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.808773    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fecce31e-a6fa-443f-aedf-327ef9710f2a-istio-podinfo\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq\" (UID: \"fecce31e-a6fa-443f-aedf-327ef9710f2a\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq"
Oct 30 05:00:21.808889 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.808809    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e29899b-8f40-4c21-a63b-2ce27d602dc3-credential-socket\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft\" (UID: \"7e29899b-8f40-4c21-a63b-2ce27d602dc3\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft"
Oct 30 05:00:21.808889 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.808841    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fecce31e-a6fa-443f-aedf-327ef9710f2a-workload-socket\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq\" (UID: \"fecce31e-a6fa-443f-aedf-327ef9710f2a\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq"
Oct 30 05:00:21.809113 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.808971    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82219032-2bf7-4bc8-89d3-5063a7ab1f0f-maas-api-openshift-ca" pod "82219032-2bf7-4bc8-89d3-5063a7ab1f0f" (UID: "82219032-2bf7-4bc8-89d3-5063a7ab1f0f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:00:21.809113 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.808974    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e29899b-8f40-4c21-a63b-2ce27d602dc3-istio-data\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft\" (UID: \"7e29899b-8f40-4c21-a63b-2ce27d602dc3\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft"
Oct 30 05:00:21.809113 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.809054    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fecce31e-a6fa-443f-aedf-327ef9710f2a-istio-data\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq\" (UID: \"fecce31e-a6fa-443f-aedf-327ef9710f2a\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq"
Oct 30 05:00:21.809265 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.809173    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e29899b-8f40-4c21-a63b-2ce27d602dc3-istio-token\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft\" (UID: \"7e29899b-8f40-4c21-a63b-2ce27d602dc3\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft"
Oct 30 05:00:21.809265 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.809218    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8djx\" (UniqueName: \"kubernetes.io/projected/fecce31e-a6fa-443f-aedf-327ef9710f2a-kube-api-access-l8djx\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq\" (UID: \"fecce31e-a6fa-443f-aedf-327ef9710f2a\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq"
Oct 30 05:00:21.809265 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.809248    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e29899b-8f40-4c21-a63b-2ce27d602dc3-workload-certs\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft\" (UID: \"7e29899b-8f40-4c21-a63b-2ce27d602dc3\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft"
Oct 30 05:00:21.809417 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.809274    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e29899b-8f40-4c21-a63b-2ce27d602dc3-istio-envoy\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft\" (UID: \"7e29899b-8f40-4c21-a63b-2ce27d602dc3\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft"
Oct 30 05:00:21.809417 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.809329    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fecce31e-a6fa-443f-aedf-327ef9710f2a-workload-certs\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq\" (UID: \"fecce31e-a6fa-443f-aedf-327ef9710f2a\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq"
Oct 30 05:00:21.809417 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.809360    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e29899b-8f40-4c21-a63b-2ce27d602dc3-istiod-ca-cert\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft\" (UID: \"7e29899b-8f40-4c21-a63b-2ce27d602dc3\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft"
Oct 30 05:00:21.809592 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.809567    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fecce31e-a6fa-443f-aedf-327ef9710f2a-istiod-ca-cert\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq\" (UID: \"fecce31e-a6fa-443f-aedf-327ef9710f2a\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq"
Oct 30 05:00:21.809692 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.809611    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fecce31e-a6fa-443f-aedf-327ef9710f2a-istio-token\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq\" (UID: \"fecce31e-a6fa-443f-aedf-327ef9710f2a\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq"
Oct 30 05:00:21.809692 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.809657    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e29899b-8f40-4c21-a63b-2ce27d602dc3-istio-podinfo\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft\" (UID: \"7e29899b-8f40-4c21-a63b-2ce27d602dc3\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft"
Oct 30 05:00:21.809770 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.809712    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fecce31e-a6fa-443f-aedf-327ef9710f2a-credential-socket\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq\" (UID: \"fecce31e-a6fa-443f-aedf-327ef9710f2a\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq"
Oct 30 05:00:21.809770 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.809756    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxggk\" (UniqueName: \"kubernetes.io/projected/7e29899b-8f40-4c21-a63b-2ce27d602dc3-kube-api-access-xxggk\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft\" (UID: \"7e29899b-8f40-4c21-a63b-2ce27d602dc3\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft"
Oct 30 05:00:21.809861 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.809835    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fecce31e-a6fa-443f-aedf-327ef9710f2a-istio-envoy\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq\" (UID: \"fecce31e-a6fa-443f-aedf-327ef9710f2a\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq"
Oct 30 05:00:21.809861 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.809853    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e29899b-8f40-4c21-a63b-2ce27d602dc3-workload-socket\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft\" (UID: \"7e29899b-8f40-4c21-a63b-2ce27d602dc3\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft"
Oct 30 05:00:21.809961 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.809929    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82219032-2bf7-4bc8-89d3-5063a7ab1f0f-maas-api-openshift-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:00:21.810815 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.810795    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82219032-2bf7-4bc8-89d3-5063a7ab1f0f-maas-api-xks-ca" pod "82219032-2bf7-4bc8-89d3-5063a7ab1f0f" (UID: "82219032-2bf7-4bc8-89d3-5063a7ab1f0f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:00:21.811021 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.810997    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82219032-2bf7-4bc8-89d3-5063a7ab1f0f-kube-api-access-bqb5s" pod "82219032-2bf7-4bc8-89d3-5063a7ab1f0f" (UID: "82219032-2bf7-4bc8-89d3-5063a7ab1f0f"). InnerVolumeSpecName "kube-api-access-bqb5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:00:21.911347 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.911252    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8djx\" (UniqueName: \"kubernetes.io/projected/fecce31e-a6fa-443f-aedf-327ef9710f2a-kube-api-access-l8djx\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq\" (UID: \"fecce31e-a6fa-443f-aedf-327ef9710f2a\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq"
Oct 30 05:00:21.911347 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.911298    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e29899b-8f40-4c21-a63b-2ce27d602dc3-workload-certs\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft\" (UID: \"7e29899b-8f40-4c21-a63b-2ce27d602dc3\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft"
Oct 30 05:00:21.911347 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.911323    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e29899b-8f40-4c21-a63b-2ce27d602dc3-istio-envoy\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft\" (UID: \"7e29899b-8f40-4c21-a63b-2ce27d602dc3\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft"
Oct 30 05:00:21.911626 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.911388    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fecce31e-a6fa-443f-aedf-327ef9710f2a-workload-certs\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq\" (UID: \"fecce31e-a6fa-443f-aedf-327ef9710f2a\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq"
Oct 30 05:00:21.911626 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.911436    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e29899b-8f40-4c21-a63b-2ce27d602dc3-istiod-ca-cert\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft\" (UID: \"7e29899b-8f40-4c21-a63b-2ce27d602dc3\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft"
Oct 30 05:00:21.911626 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.911494    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fecce31e-a6fa-443f-aedf-327ef9710f2a-istiod-ca-cert\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq\" (UID: \"fecce31e-a6fa-443f-aedf-327ef9710f2a\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq"
Oct 30 05:00:21.911626 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.911521    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fecce31e-a6fa-443f-aedf-327ef9710f2a-istio-token\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq\" (UID: \"fecce31e-a6fa-443f-aedf-327ef9710f2a\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq"
Oct 30 05:00:21.911626 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.911544    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e29899b-8f40-4c21-a63b-2ce27d602dc3-istio-podinfo\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft\" (UID: \"7e29899b-8f40-4c21-a63b-2ce27d602dc3\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft"
Oct 30 05:00:21.911626 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.911575    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fecce31e-a6fa-443f-aedf-327ef9710f2a-credential-socket\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq\" (UID: \"fecce31e-a6fa-443f-aedf-327ef9710f2a\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq"
Oct 30 05:00:21.911927 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.911657    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxggk\" (UniqueName: \"kubernetes.io/projected/7e29899b-8f40-4c21-a63b-2ce27d602dc3-kube-api-access-xxggk\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft\" (UID: \"7e29899b-8f40-4c21-a63b-2ce27d602dc3\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft"
Oct 30 05:00:21.911927 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.911736    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fecce31e-a6fa-443f-aedf-327ef9710f2a-istio-envoy\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq\" (UID: \"fecce31e-a6fa-443f-aedf-327ef9710f2a\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq"
Oct 30 05:00:21.911927 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.911765    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e29899b-8f40-4c21-a63b-2ce27d602dc3-workload-socket\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft\" (UID: \"7e29899b-8f40-4c21-a63b-2ce27d602dc3\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft"
Oct 30 05:00:21.911927 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.911793    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fecce31e-a6fa-443f-aedf-327ef9710f2a-istio-podinfo\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq\" (UID: \"fecce31e-a6fa-443f-aedf-327ef9710f2a\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq"
Oct 30 05:00:21.911927 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.911818    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e29899b-8f40-4c21-a63b-2ce27d602dc3-credential-socket\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft\" (UID: \"7e29899b-8f40-4c21-a63b-2ce27d602dc3\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft"
Oct 30 05:00:21.911927 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.911850    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fecce31e-a6fa-443f-aedf-327ef9710f2a-workload-socket\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq\" (UID: \"fecce31e-a6fa-443f-aedf-327ef9710f2a\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq"
Oct 30 05:00:21.911927 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.911883    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e29899b-8f40-4c21-a63b-2ce27d602dc3-istio-data\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft\" (UID: \"7e29899b-8f40-4c21-a63b-2ce27d602dc3\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft"
Oct 30 05:00:21.912235 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.911946    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fecce31e-a6fa-443f-aedf-327ef9710f2a-istio-data\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq\" (UID: \"fecce31e-a6fa-443f-aedf-327ef9710f2a\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq"
Oct 30 05:00:21.912235 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.912042    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e29899b-8f40-4c21-a63b-2ce27d602dc3-istio-token\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft\" (UID: \"7e29899b-8f40-4c21-a63b-2ce27d602dc3\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft"
Oct 30 05:00:21.912235 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.912108    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqb5s\" (UniqueName: \"kubernetes.io/projected/82219032-2bf7-4bc8-89d3-5063a7ab1f0f-kube-api-access-bqb5s\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:00:21.912235 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.912125    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82219032-2bf7-4bc8-89d3-5063a7ab1f0f-maas-api-xks-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:00:21.912535 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.912495    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fecce31e-a6fa-443f-aedf-327ef9710f2a-workload-certs\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq\" (UID: \"fecce31e-a6fa-443f-aedf-327ef9710f2a\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq"
Oct 30 05:00:21.912621 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.912543    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e29899b-8f40-4c21-a63b-2ce27d602dc3-workload-certs\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft\" (UID: \"7e29899b-8f40-4c21-a63b-2ce27d602dc3\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft"
Oct 30 05:00:21.913114 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.913085    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e29899b-8f40-4c21-a63b-2ce27d602dc3-istiod-ca-cert\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft\" (UID: \"7e29899b-8f40-4c21-a63b-2ce27d602dc3\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft"
Oct 30 05:00:21.913273 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.913159    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e29899b-8f40-4c21-a63b-2ce27d602dc3-istio-data\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft\" (UID: \"7e29899b-8f40-4c21-a63b-2ce27d602dc3\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft"
Oct 30 05:00:21.913362 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.913339    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fecce31e-a6fa-443f-aedf-327ef9710f2a-istio-data\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq\" (UID: \"fecce31e-a6fa-443f-aedf-327ef9710f2a\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq"
Oct 30 05:00:21.913453 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.913387    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fecce31e-a6fa-443f-aedf-327ef9710f2a-workload-socket\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq\" (UID: \"fecce31e-a6fa-443f-aedf-327ef9710f2a\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq"
Oct 30 05:00:21.913588 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.913567    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e29899b-8f40-4c21-a63b-2ce27d602dc3-workload-socket\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft\" (UID: \"7e29899b-8f40-4c21-a63b-2ce27d602dc3\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft"
Oct 30 05:00:21.913834 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.913812    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fecce31e-a6fa-443f-aedf-327ef9710f2a-credential-socket\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq\" (UID: \"fecce31e-a6fa-443f-aedf-327ef9710f2a\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq"
Oct 30 05:00:21.914416 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.914388    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fecce31e-a6fa-443f-aedf-327ef9710f2a-istiod-ca-cert\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq\" (UID: \"fecce31e-a6fa-443f-aedf-327ef9710f2a\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq"
Oct 30 05:00:21.914748 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.914721    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e29899b-8f40-4c21-a63b-2ce27d602dc3-credential-socket\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft\" (UID: \"7e29899b-8f40-4c21-a63b-2ce27d602dc3\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft"
Oct 30 05:00:21.915246 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.915222    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e29899b-8f40-4c21-a63b-2ce27d602dc3-istio-envoy\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft\" (UID: \"7e29899b-8f40-4c21-a63b-2ce27d602dc3\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft"
Oct 30 05:00:21.915358 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.915265    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fecce31e-a6fa-443f-aedf-327ef9710f2a-istio-envoy\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq\" (UID: \"fecce31e-a6fa-443f-aedf-327ef9710f2a\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq"
Oct 30 05:00:21.916427 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.916399    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fecce31e-a6fa-443f-aedf-327ef9710f2a-istio-podinfo\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq\" (UID: \"fecce31e-a6fa-443f-aedf-327ef9710f2a\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq"
Oct 30 05:00:21.917013 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.916989    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e29899b-8f40-4c21-a63b-2ce27d602dc3-istio-podinfo\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft\" (UID: \"7e29899b-8f40-4c21-a63b-2ce27d602dc3\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft"
Oct 30 05:00:21.929832 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.929809    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8djx\" (UniqueName: \"kubernetes.io/projected/fecce31e-a6fa-443f-aedf-327ef9710f2a-kube-api-access-l8djx\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq\" (UID: \"fecce31e-a6fa-443f-aedf-327ef9710f2a\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq"
Oct 30 05:00:21.930199 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.930176    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e29899b-8f40-4c21-a63b-2ce27d602dc3-istio-token\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft\" (UID: \"7e29899b-8f40-4c21-a63b-2ce27d602dc3\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft"
Oct 30 05:00:21.931341 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.931322    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fecce31e-a6fa-443f-aedf-327ef9710f2a-istio-token\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq\" (UID: \"fecce31e-a6fa-443f-aedf-327ef9710f2a\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq"
Oct 30 05:00:21.931593 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:21.931576    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxggk\" (UniqueName: \"kubernetes.io/projected/7e29899b-8f40-4c21-a63b-2ce27d602dc3-kube-api-access-xxggk\") pod \"e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft\" (UID: \"7e29899b-8f40-4c21-a63b-2ce27d602dc3\") " pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft"
Oct 30 05:00:22.021974 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:22.021949    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft"
Oct 30 05:00:22.069297 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:22.069266    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq"
Oct 30 05:00:22.333438 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:22.333410    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft"]
Oct 30 05:00:22.334270 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:00:22.334245    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e29899b_8f40_4c21_a63b_2ce27d602dc3.slice/crio-4dd764ea64cbb6bf41c8a339dedf44048fce5f3b7691ab5ef277dd96767d307f WatchSource:0}: Error finding container 4dd764ea64cbb6bf41c8a339dedf44048fce5f3b7691ab5ef277dd96767d307f: Status 404 returned error can't find the container with id 4dd764ea64cbb6bf41c8a339dedf44048fce5f3b7691ab5ef277dd96767d307f
Oct 30 05:00:22.336786 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:22.336749    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:00:22.336884 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:22.336826    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:00:22.336884 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:22.336870    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:00:22.387940 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:22.387915    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq"]
Oct 30 05:00:22.388286 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:00:22.388265    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfecce31e_a6fa_443f_aedf_327ef9710f2a.slice/crio-574c55e467113f34a19fc72828fcf6abdf63b2b3787e1dee02ea019ca907f3a1 WatchSource:0}: Error finding container 574c55e467113f34a19fc72828fcf6abdf63b2b3787e1dee02ea019ca907f3a1: Status 404 returned error can't find the container with id 574c55e467113f34a19fc72828fcf6abdf63b2b3787e1dee02ea019ca907f3a1
Oct 30 05:00:22.391207 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:22.391179    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:00:22.391295 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:22.391239    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:00:22.391295 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:22.391270    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:00:22.633183 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:22.633096    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e92d9e6-ctg5b" event={"ID":"82219032-2bf7-4bc8-89d3-5063a7ab1f0f","Type":"ContainerDied","Data":"f9dda853127dbbad16518da4a25a3e4bf155efb8206407ae02283ac28e7cf60e"}
Oct 30 05:00:22.633183 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:22.633118    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e92d9e6-ctg5b"
Oct 30 05:00:22.633183 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:22.633151    5898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9dda853127dbbad16518da4a25a3e4bf155efb8206407ae02283ac28e7cf60e"
Oct 30 05:00:22.634784 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:22.634760    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq" event={"ID":"fecce31e-a6fa-443f-aedf-327ef9710f2a","Type":"ContainerStarted","Data":"706fa53890f44692b093fe30d77e461d56d14e307071755ea21e09e28bf2d70b"}
Oct 30 05:00:22.634921 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:22.634799    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq" event={"ID":"fecce31e-a6fa-443f-aedf-327ef9710f2a","Type":"ContainerStarted","Data":"574c55e467113f34a19fc72828fcf6abdf63b2b3787e1dee02ea019ca907f3a1"}
Oct 30 05:00:22.636622 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:22.636596    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft" event={"ID":"7e29899b-8f40-4c21-a63b-2ce27d602dc3","Type":"ContainerStarted","Data":"6555b1b0099159bd01fd8616bb1280319a0e9e03a6133b9614046d590e1f477b"}
Oct 30 05:00:22.636737 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:22.636637    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft" event={"ID":"7e29899b-8f40-4c21-a63b-2ce27d602dc3","Type":"ContainerStarted","Data":"4dd764ea64cbb6bf41c8a339dedf44048fce5f3b7691ab5ef277dd96767d307f"}
Oct 30 05:00:22.678027 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:22.677967    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq" podStartSLOduration=1.6779472800000002 podStartE2EDuration="1.67794728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:22.67652827 +0000 UTC m=+951.900348190" watchObservedRunningTime="2026-10-30 05:00:22.67794728 +0000 UTC m=+951.901767184"
Oct 30 05:00:22.737484 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:22.737433    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft" podStartSLOduration=1.737418358 podStartE2EDuration="1.737418358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:22.734187894 +0000 UTC m=+951.958007810" watchObservedRunningTime="2026-10-30 05:00:22.737418358 +0000 UTC m=+951.961238258"
Oct 30 05:00:22.874253 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:22.874215    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:00:23.025005 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:23.024336    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft"
Oct 30 05:00:23.032927 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:23.031195    5898 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 05:00:23.032927 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:23.031263    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft" podUID="7e29899b-8f40-4c21-a63b-2ce27d602dc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 05:00:23.070490 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:23.070458    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq"
Oct 30 05:00:23.071963 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:23.071937    5898 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 05:00:23.072067 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:23.071991    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq" podUID="fecce31e-a6fa-443f-aedf-327ef9710f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 05:00:23.216922 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:23.215798    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:00:24.036924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:24.036325    5898 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 05:00:24.036924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:24.036410    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft" podUID="7e29899b-8f40-4c21-a63b-2ce27d602dc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 05:00:24.070210 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:24.070176    5898 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 05:00:24.070375 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:24.070222    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq" podUID="fecce31e-a6fa-443f-aedf-327ef9710f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 05:00:24.947925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:24.942555    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:00:25.027809 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:25.027319    5898 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 05:00:25.027809 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:25.027401    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft" podUID="7e29899b-8f40-4c21-a63b-2ce27d602dc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 05:00:25.078355 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:25.078316    5898 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 05:00:25.079009 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:25.078976    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq" podUID="fecce31e-a6fa-443f-aedf-327ef9710f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 05:00:25.566926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:25.566793    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:00:26.041926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:26.031296    5898 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 05:00:26.041926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:26.031373    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft" podUID="7e29899b-8f40-4c21-a63b-2ce27d602dc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 05:00:26.075925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:26.070888    5898 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 05:00:26.075925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:26.070964    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq" podUID="fecce31e-a6fa-443f-aedf-327ef9710f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 05:00:26.293979 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:26.289999    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49" podUID="05eef4eb-f379-4dc7-a947-123bd85a4968" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:00:26.417968 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:26.411330    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx" podUID="d523008f-c740-4a41-89cb-e82e1ffa0094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:00:26.448927 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:26.427976    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58" podUID="c27f3b21-1447-4af9-8d99-c04fa9b23cda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:00:26.448927 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:26.439032    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9" podUID="1647f909-5580-4219-a2e1-30b4cd994379" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:00:26.475922 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:26.475209    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb" podUID="6f16bd6a-a7f0-49bb-844f-57b777bde2ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 05:00:26.568102 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:26.567980    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849" podUID="a9707f50-d5e1-409f-b8f3-c58629677aaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 05:00:26.580935 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:26.578981    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247" podUID="1d865d3e-a6e9-407e-844e-cc75fc7f1354" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:00:26.625026 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:26.624978    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5" podUID="ebf8f4a2-8ac6-48ec-bdba-3f6a99653264" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 05:00:26.626335 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:26.626306    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g" podUID="70294a85-1050-4632-b3bf-c3e44a4c0b41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 05:00:27.039149 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:27.037840    5898 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 05:00:27.039149 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:27.037928    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft" podUID="7e29899b-8f40-4c21-a63b-2ce27d602dc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 05:00:27.108208 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:27.108174    5898 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 05:00:27.123042 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:27.108438    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq" podUID="fecce31e-a6fa-443f-aedf-327ef9710f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 05:00:27.234163 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:00:27.234120    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:00:27.240313 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:27.240270    5898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58" podUID="c27f3b21-1447-4af9-8d99-c04fa9b23cda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:00:27.271050 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:27.265004    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6" podUID="b09804e3-8135-4f6d-ae34-c45d6da9728e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:00:27.293028 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:27.291009    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49" podUID="05eef4eb-f379-4dc7-a947-123bd85a4968" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:00:27.293028 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:27.291450    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49"
Oct 30 05:00:27.462958 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:27.443660    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx" podUID="d523008f-c740-4a41-89cb-e82e1ffa0094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 05:00:27.463799 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:27.454995    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9" podUID="1647f909-5580-4219-a2e1-30b4cd994379" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:00:27.489419 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:27.464029    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx"
Oct 30 05:00:27.489717 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:27.489698    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9"
Oct 30 05:00:27.514145 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:27.489031    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb" podUID="6f16bd6a-a7f0-49bb-844f-57b777bde2ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:00:27.514684 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:27.514662    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb"
Oct 30 05:00:27.570123 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:27.570025    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849" podUID="a9707f50-d5e1-409f-b8f3-c58629677aaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 05:00:27.624960 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:27.612463    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849"
Oct 30 05:00:27.648935 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:27.641864    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g" podUID="70294a85-1050-4632-b3bf-c3e44a4c0b41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:00:27.648935 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:27.642230    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5" podUID="ebf8f4a2-8ac6-48ec-bdba-3f6a99653264" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 05:00:27.648935 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:27.642490    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g"
Oct 30 05:00:27.648935 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:27.642544    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5"
Oct 30 05:00:27.981926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:27.939927    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:00:27.985641 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:27.982228    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5"
Oct 30 05:00:28.031396 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:28.031362    5898 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 05:00:28.031699 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:28.031655    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft" podUID="7e29899b-8f40-4c21-a63b-2ce27d602dc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 05:00:28.116165 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:28.116127    5898 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 05:00:28.124122 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:28.124084    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq" podUID="fecce31e-a6fa-443f-aedf-327ef9710f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 05:00:28.350536 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:28.350498    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49" podUID="05eef4eb-f379-4dc7-a947-123bd85a4968" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 05:00:28.446150 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:28.446101    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58" podUID="c27f3b21-1447-4af9-8d99-c04fa9b23cda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:00:28.467566 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:28.467513    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx" podUID="d523008f-c740-4a41-89cb-e82e1ffa0094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 05:00:28.495181 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:28.495104    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9" podUID="1647f909-5580-4219-a2e1-30b4cd994379" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 05:00:28.575936 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:28.572832    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh" podUID="38428a53-04c7-400a-9c78-52407e982478" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:00:28.575936 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:28.573832    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb" podUID="6f16bd6a-a7f0-49bb-844f-57b777bde2ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 05:00:28.576739 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:28.576704    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:00:28.592042 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:28.587099    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247" podUID="1d865d3e-a6e9-407e-844e-cc75fc7f1354" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 05:00:28.620683 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:28.620590    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849" podUID="a9707f50-d5e1-409f-b8f3-c58629677aaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:00:28.645068 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:28.645017    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g" podUID="70294a85-1050-4632-b3bf-c3e44a4c0b41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:00:28.943848 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:28.942814    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5" podUID="ebf8f4a2-8ac6-48ec-bdba-3f6a99653264" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:00:28.971278 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:28.971245    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx"
Oct 30 05:00:29.065925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:29.063488    5898 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 05:00:29.065925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:29.063542    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft" podUID="7e29899b-8f40-4c21-a63b-2ce27d602dc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 05:00:29.084926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:29.072669    5898 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 05:00:29.084926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:29.072735    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq" podUID="fecce31e-a6fa-443f-aedf-327ef9710f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 05:00:29.112926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:29.103175    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g"
Oct 30 05:00:29.149500 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:29.149471    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9"
Oct 30 05:00:29.173595 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:29.173567    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849"
Oct 30 05:00:29.263183 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:29.263099    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb"
Oct 30 05:00:29.266979 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:29.266928    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49"
Oct 30 05:00:30.023135 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:30.023099    5898 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 05:00:30.023638 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:30.023152    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft" podUID="7e29899b-8f40-4c21-a63b-2ce27d602dc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 05:00:30.071049 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:30.070995    5898 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 05:00:30.071244 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:30.071063    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq" podUID="fecce31e-a6fa-443f-aedf-327ef9710f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 05:00:30.549326 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:30.549286    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:00:32.041491 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:32.041449    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq"
Oct 30 05:00:32.042025 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:32.041932    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq"
Oct 30 05:00:32.043642 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:32.043622    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq"
Oct 30 05:00:32.064012 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:32.063989    5898 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 05:00:32.064136 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:32.064030    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft" podUID="7e29899b-8f40-4c21-a63b-2ce27d602dc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:00:32.344198 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:32.344167    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft"
Oct 30 05:00:32.344971 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:32.344942    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:00:32.345095 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:32.345021    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft"
Oct 30 05:00:32.345843 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:32.345821    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft"
Oct 30 05:00:34.037640 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:34.037611    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:00:35.787895 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:35.787856    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:00:37.519225 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:37.519192    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:00:38.407621 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:38.407580    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:00:39.200516 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:39.200467    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:00:40.458346 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:40.458294    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:00:41.535045 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:41.534998    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx" podUID="d523008f-c740-4a41-89cb-e82e1ffa0094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:00:41.630949 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:41.630880    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849" podUID="a9707f50-d5e1-409f-b8f3-c58629677aaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 05:00:41.941977 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:41.941924    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5" podUID="ebf8f4a2-8ac6-48ec-bdba-3f6a99653264" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:00:42.110025 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:42.109979    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g" podUID="70294a85-1050-4632-b3bf-c3e44a4c0b41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 05:00:42.153240 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:42.153201    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9" podUID="1647f909-5580-4219-a2e1-30b4cd994379" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:00:42.578045 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:42.578001    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:00:44.630236 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:44.630193    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:00:46.239486 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:46.239451    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:00:48.176042 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:48.176011    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:00:49.914939 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:49.914884    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6"]
Oct 30 05:00:49.915657 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:49.915631    5898 cpu_manager.go:414] "containerMap: removing container" podUID="82219032-2bf7-4bc8-89d3-5063a7ab1f0f" containerName="revoke-keys"
Oct 30 05:00:49.915657 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:49.915657    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82219032-2bf7-4bc8-89d3-5063a7ab1f0f" containerName="revoke-keys"
Oct 30 05:00:49.915876 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:49.915862    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="82219032-2bf7-4bc8-89d3-5063a7ab1f0f" containerName="revoke-keys"
Oct 30 05:00:49.919527 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:49.919509    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6"
Oct 30 05:00:49.936156 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:49.936130    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-397179\"/\"distinct-w2-397179-kserve-self-signed-certs\""
Oct 30 05:00:49.937169 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:49.937148    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6"]
Oct 30 05:00:49.945670 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:49.945615    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:00:50.043875 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:50.043843    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47ff272a-ac5c-4f59-bf7b-5c0692d09386-home\") pod \"distinct-w2-397179-kserve-56547fb455-flrg6\" (UID: \"47ff272a-ac5c-4f59-bf7b-5c0692d09386\") " pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6"
Oct 30 05:00:50.044049 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:50.043883    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvgtk\" (UniqueName: \"kubernetes.io/projected/47ff272a-ac5c-4f59-bf7b-5c0692d09386-kube-api-access-bvgtk\") pod \"distinct-w2-397179-kserve-56547fb455-flrg6\" (UID: \"47ff272a-ac5c-4f59-bf7b-5c0692d09386\") " pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6"
Oct 30 05:00:50.044049 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:50.043989    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47ff272a-ac5c-4f59-bf7b-5c0692d09386-dshm\") pod \"distinct-w2-397179-kserve-56547fb455-flrg6\" (UID: \"47ff272a-ac5c-4f59-bf7b-5c0692d09386\") " pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6"
Oct 30 05:00:50.044132 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:50.044114    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47ff272a-ac5c-4f59-bf7b-5c0692d09386-tls-certs\") pod \"distinct-w2-397179-kserve-56547fb455-flrg6\" (UID: \"47ff272a-ac5c-4f59-bf7b-5c0692d09386\") " pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6"
Oct 30 05:00:50.044292 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:50.044267    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47ff272a-ac5c-4f59-bf7b-5c0692d09386-model-cache\") pod \"distinct-w2-397179-kserve-56547fb455-flrg6\" (UID: \"47ff272a-ac5c-4f59-bf7b-5c0692d09386\") " pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6"
Oct 30 05:00:50.044380 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:50.044336    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47ff272a-ac5c-4f59-bf7b-5c0692d09386-tmp-dir\") pod \"distinct-w2-397179-kserve-56547fb455-flrg6\" (UID: \"47ff272a-ac5c-4f59-bf7b-5c0692d09386\") " pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6"
Oct 30 05:00:50.081167 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:50.081133    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:00:50.145425 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:50.145384    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47ff272a-ac5c-4f59-bf7b-5c0692d09386-home\") pod \"distinct-w2-397179-kserve-56547fb455-flrg6\" (UID: \"47ff272a-ac5c-4f59-bf7b-5c0692d09386\") " pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6"
Oct 30 05:00:50.145687 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:50.145668    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvgtk\" (UniqueName: \"kubernetes.io/projected/47ff272a-ac5c-4f59-bf7b-5c0692d09386-kube-api-access-bvgtk\") pod \"distinct-w2-397179-kserve-56547fb455-flrg6\" (UID: \"47ff272a-ac5c-4f59-bf7b-5c0692d09386\") " pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6"
Oct 30 05:00:50.145811 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:50.145797    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47ff272a-ac5c-4f59-bf7b-5c0692d09386-dshm\") pod \"distinct-w2-397179-kserve-56547fb455-flrg6\" (UID: \"47ff272a-ac5c-4f59-bf7b-5c0692d09386\") " pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6"
Oct 30 05:00:50.145985 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:50.145966    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47ff272a-ac5c-4f59-bf7b-5c0692d09386-tls-certs\") pod \"distinct-w2-397179-kserve-56547fb455-flrg6\" (UID: \"47ff272a-ac5c-4f59-bf7b-5c0692d09386\") " pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6"
Oct 30 05:00:50.146204 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:50.146187    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47ff272a-ac5c-4f59-bf7b-5c0692d09386-model-cache\") pod \"distinct-w2-397179-kserve-56547fb455-flrg6\" (UID: \"47ff272a-ac5c-4f59-bf7b-5c0692d09386\") " pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6"
Oct 30 05:00:50.146312 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:50.146300    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47ff272a-ac5c-4f59-bf7b-5c0692d09386-tmp-dir\") pod \"distinct-w2-397179-kserve-56547fb455-flrg6\" (UID: \"47ff272a-ac5c-4f59-bf7b-5c0692d09386\") " pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6"
Oct 30 05:00:50.148134 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:50.146797    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47ff272a-ac5c-4f59-bf7b-5c0692d09386-home\") pod \"distinct-w2-397179-kserve-56547fb455-flrg6\" (UID: \"47ff272a-ac5c-4f59-bf7b-5c0692d09386\") " pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6"
Oct 30 05:00:50.148325 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:50.147444    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47ff272a-ac5c-4f59-bf7b-5c0692d09386-model-cache\") pod \"distinct-w2-397179-kserve-56547fb455-flrg6\" (UID: \"47ff272a-ac5c-4f59-bf7b-5c0692d09386\") " pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6"
Oct 30 05:00:50.148463 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:50.148001    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47ff272a-ac5c-4f59-bf7b-5c0692d09386-tmp-dir\") pod \"distinct-w2-397179-kserve-56547fb455-flrg6\" (UID: \"47ff272a-ac5c-4f59-bf7b-5c0692d09386\") " pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6"
Oct 30 05:00:50.157284 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:50.157259    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47ff272a-ac5c-4f59-bf7b-5c0692d09386-dshm\") pod \"distinct-w2-397179-kserve-56547fb455-flrg6\" (UID: \"47ff272a-ac5c-4f59-bf7b-5c0692d09386\") " pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6"
Oct 30 05:00:50.167722 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:50.167639    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvgtk\" (UniqueName: \"kubernetes.io/projected/47ff272a-ac5c-4f59-bf7b-5c0692d09386-kube-api-access-bvgtk\") pod \"distinct-w2-397179-kserve-56547fb455-flrg6\" (UID: \"47ff272a-ac5c-4f59-bf7b-5c0692d09386\") " pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6"
Oct 30 05:00:50.171363 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:50.171337    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47ff272a-ac5c-4f59-bf7b-5c0692d09386-tls-certs\") pod \"distinct-w2-397179-kserve-56547fb455-flrg6\" (UID: \"47ff272a-ac5c-4f59-bf7b-5c0692d09386\") " pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6"
Oct 30 05:00:50.237761 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:50.237490    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6"
Oct 30 05:00:50.552281 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:50.552253    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6"]
Oct 30 05:00:50.554526 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:00:50.554494    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod47ff272a_ac5c_4f59_bf7b_5c0692d09386.slice/crio-24e7dced4f47653ec987bd2fe8dd3c5c5149650c88af1ce06d5a282a295efe81 WatchSource:0}: Error finding container 24e7dced4f47653ec987bd2fe8dd3c5c5149650c88af1ce06d5a282a295efe81: Status 404 returned error can't find the container with id 24e7dced4f47653ec987bd2fe8dd3c5c5149650c88af1ce06d5a282a295efe81
Oct 30 05:00:51.558210 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:51.558173    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6" event={"ID":"47ff272a-ac5c-4f59-bf7b-5c0692d09386","Type":"ContainerStarted","Data":"5ea897a4d77a647b44003d22b196cc3136b848404e7f753cb8fd5d8a9a4bd7c8"}
Oct 30 05:00:51.558210 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:51.558213    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6" event={"ID":"47ff272a-ac5c-4f59-bf7b-5c0692d09386","Type":"ContainerStarted","Data":"24e7dced4f47653ec987bd2fe8dd3c5c5149650c88af1ce06d5a282a295efe81"}
Oct 30 05:00:51.595614 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:51.595559    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6" podStartSLOduration=2.595546955 podStartE2EDuration="2.595546955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:51.59460659 +0000 UTC m=+980.818426511" watchObservedRunningTime="2026-10-30 05:00:51.595546955 +0000 UTC m=+980.819366855"
Oct 30 05:00:51.646218 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:51.646158    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:00:52.176523 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.176487    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp"]
Oct 30 05:00:52.180510 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.180494    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp"
Oct 30 05:00:52.201324 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.201294    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp"]
Oct 30 05:00:52.238876 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.238847    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6"
Oct 30 05:00:52.272858 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.272826    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e97ea58e-dd00-4d95-b178-806306ff635c-workload-socket\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp\" (UID: \"e97ea58e-dd00-4d95-b178-806306ff635c\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp"
Oct 30 05:00:52.275078 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.272922    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e97ea58e-dd00-4d95-b178-806306ff635c-istiod-ca-cert\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp\" (UID: \"e97ea58e-dd00-4d95-b178-806306ff635c\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp"
Oct 30 05:00:52.275078 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.272964    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e97ea58e-dd00-4d95-b178-806306ff635c-istio-data\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp\" (UID: \"e97ea58e-dd00-4d95-b178-806306ff635c\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp"
Oct 30 05:00:52.275078 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.273090    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e97ea58e-dd00-4d95-b178-806306ff635c-istio-envoy\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp\" (UID: \"e97ea58e-dd00-4d95-b178-806306ff635c\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp"
Oct 30 05:00:52.275078 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.273130    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e97ea58e-dd00-4d95-b178-806306ff635c-workload-certs\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp\" (UID: \"e97ea58e-dd00-4d95-b178-806306ff635c\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp"
Oct 30 05:00:52.275078 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.273150    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e97ea58e-dd00-4d95-b178-806306ff635c-istio-podinfo\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp\" (UID: \"e97ea58e-dd00-4d95-b178-806306ff635c\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp"
Oct 30 05:00:52.275078 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.273167    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5vkx\" (UniqueName: \"kubernetes.io/projected/e97ea58e-dd00-4d95-b178-806306ff635c-kube-api-access-w5vkx\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp\" (UID: \"e97ea58e-dd00-4d95-b178-806306ff635c\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp"
Oct 30 05:00:52.275078 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.273228    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e97ea58e-dd00-4d95-b178-806306ff635c-istio-token\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp\" (UID: \"e97ea58e-dd00-4d95-b178-806306ff635c\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp"
Oct 30 05:00:52.275078 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.273278    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e97ea58e-dd00-4d95-b178-806306ff635c-credential-socket\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp\" (UID: \"e97ea58e-dd00-4d95-b178-806306ff635c\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp"
Oct 30 05:00:52.293932 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.293893    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6"]
Oct 30 05:00:52.298132 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.298105    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6"
Oct 30 05:00:52.319966 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.319927    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6"]
Oct 30 05:00:52.374302 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.374269    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e97ea58e-dd00-4d95-b178-806306ff635c-istio-data\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp\" (UID: \"e97ea58e-dd00-4d95-b178-806306ff635c\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp"
Oct 30 05:00:52.374461 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.374316    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79faca01-0058-4ef5-ab44-5bb52d1c3bb0-istiod-ca-cert\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6\" (UID: \"79faca01-0058-4ef5-ab44-5bb52d1c3bb0\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6"
Oct 30 05:00:52.374461 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.374354    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79faca01-0058-4ef5-ab44-5bb52d1c3bb0-workload-socket\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6\" (UID: \"79faca01-0058-4ef5-ab44-5bb52d1c3bb0\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6"
Oct 30 05:00:52.374461 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.374382    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79faca01-0058-4ef5-ab44-5bb52d1c3bb0-credential-socket\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6\" (UID: \"79faca01-0058-4ef5-ab44-5bb52d1c3bb0\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6"
Oct 30 05:00:52.374575 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.374520    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e97ea58e-dd00-4d95-b178-806306ff635c-istio-envoy\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp\" (UID: \"e97ea58e-dd00-4d95-b178-806306ff635c\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp"
Oct 30 05:00:52.374575 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.374558    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w95rq\" (UniqueName: \"kubernetes.io/projected/79faca01-0058-4ef5-ab44-5bb52d1c3bb0-kube-api-access-w95rq\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6\" (UID: \"79faca01-0058-4ef5-ab44-5bb52d1c3bb0\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6"
Oct 30 05:00:52.374641 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.374585    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e97ea58e-dd00-4d95-b178-806306ff635c-workload-certs\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp\" (UID: \"e97ea58e-dd00-4d95-b178-806306ff635c\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp"
Oct 30 05:00:52.374641 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.374626    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79faca01-0058-4ef5-ab44-5bb52d1c3bb0-istio-podinfo\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6\" (UID: \"79faca01-0058-4ef5-ab44-5bb52d1c3bb0\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6"
Oct 30 05:00:52.374706 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.374649    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e97ea58e-dd00-4d95-b178-806306ff635c-istio-data\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp\" (UID: \"e97ea58e-dd00-4d95-b178-806306ff635c\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp"
Oct 30 05:00:52.374765 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.374744    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e97ea58e-dd00-4d95-b178-806306ff635c-istio-podinfo\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp\" (UID: \"e97ea58e-dd00-4d95-b178-806306ff635c\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp"
Oct 30 05:00:52.374809 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.374782    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5vkx\" (UniqueName: \"kubernetes.io/projected/e97ea58e-dd00-4d95-b178-806306ff635c-kube-api-access-w5vkx\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp\" (UID: \"e97ea58e-dd00-4d95-b178-806306ff635c\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp"
Oct 30 05:00:52.374876 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.374861    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e97ea58e-dd00-4d95-b178-806306ff635c-istio-token\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp\" (UID: \"e97ea58e-dd00-4d95-b178-806306ff635c\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp"
Oct 30 05:00:52.374960 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.374938    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e97ea58e-dd00-4d95-b178-806306ff635c-workload-certs\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp\" (UID: \"e97ea58e-dd00-4d95-b178-806306ff635c\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp"
Oct 30 05:00:52.374960 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.374948    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e97ea58e-dd00-4d95-b178-806306ff635c-credential-socket\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp\" (UID: \"e97ea58e-dd00-4d95-b178-806306ff635c\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp"
Oct 30 05:00:52.375083 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.374996    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79faca01-0058-4ef5-ab44-5bb52d1c3bb0-istio-envoy\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6\" (UID: \"79faca01-0058-4ef5-ab44-5bb52d1c3bb0\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6"
Oct 30 05:00:52.375083 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.375037    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79faca01-0058-4ef5-ab44-5bb52d1c3bb0-workload-certs\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6\" (UID: \"79faca01-0058-4ef5-ab44-5bb52d1c3bb0\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6"
Oct 30 05:00:52.375191 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.375118    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79faca01-0058-4ef5-ab44-5bb52d1c3bb0-istio-token\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6\" (UID: \"79faca01-0058-4ef5-ab44-5bb52d1c3bb0\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6"
Oct 30 05:00:52.375191 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.375156    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79faca01-0058-4ef5-ab44-5bb52d1c3bb0-istio-data\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6\" (UID: \"79faca01-0058-4ef5-ab44-5bb52d1c3bb0\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6"
Oct 30 05:00:52.375297 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.375249    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e97ea58e-dd00-4d95-b178-806306ff635c-credential-socket\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp\" (UID: \"e97ea58e-dd00-4d95-b178-806306ff635c\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp"
Oct 30 05:00:52.375297 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.375251    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e97ea58e-dd00-4d95-b178-806306ff635c-workload-socket\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp\" (UID: \"e97ea58e-dd00-4d95-b178-806306ff635c\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp"
Oct 30 05:00:52.375398 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.375359    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e97ea58e-dd00-4d95-b178-806306ff635c-istiod-ca-cert\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp\" (UID: \"e97ea58e-dd00-4d95-b178-806306ff635c\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp"
Oct 30 05:00:52.375557 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.375534    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e97ea58e-dd00-4d95-b178-806306ff635c-workload-socket\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp\" (UID: \"e97ea58e-dd00-4d95-b178-806306ff635c\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp"
Oct 30 05:00:52.376175 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.376153    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e97ea58e-dd00-4d95-b178-806306ff635c-istiod-ca-cert\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp\" (UID: \"e97ea58e-dd00-4d95-b178-806306ff635c\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp"
Oct 30 05:00:52.377114 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.377096    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e97ea58e-dd00-4d95-b178-806306ff635c-istio-envoy\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp\" (UID: \"e97ea58e-dd00-4d95-b178-806306ff635c\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp"
Oct 30 05:00:52.377271 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.377253    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e97ea58e-dd00-4d95-b178-806306ff635c-istio-podinfo\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp\" (UID: \"e97ea58e-dd00-4d95-b178-806306ff635c\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp"
Oct 30 05:00:52.395120 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.395091    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e97ea58e-dd00-4d95-b178-806306ff635c-istio-token\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp\" (UID: \"e97ea58e-dd00-4d95-b178-806306ff635c\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp"
Oct 30 05:00:52.395208 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.395165    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5vkx\" (UniqueName: \"kubernetes.io/projected/e97ea58e-dd00-4d95-b178-806306ff635c-kube-api-access-w5vkx\") pod \"e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp\" (UID: \"e97ea58e-dd00-4d95-b178-806306ff635c\") " pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp"
Oct 30 05:00:52.476500 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.476425    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79faca01-0058-4ef5-ab44-5bb52d1c3bb0-istio-envoy\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6\" (UID: \"79faca01-0058-4ef5-ab44-5bb52d1c3bb0\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6"
Oct 30 05:00:52.476500 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.476469    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79faca01-0058-4ef5-ab44-5bb52d1c3bb0-workload-certs\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6\" (UID: \"79faca01-0058-4ef5-ab44-5bb52d1c3bb0\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6"
Oct 30 05:00:52.476500 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.476501    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79faca01-0058-4ef5-ab44-5bb52d1c3bb0-istio-token\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6\" (UID: \"79faca01-0058-4ef5-ab44-5bb52d1c3bb0\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6"
Oct 30 05:00:52.476764 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.476535    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79faca01-0058-4ef5-ab44-5bb52d1c3bb0-istio-data\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6\" (UID: \"79faca01-0058-4ef5-ab44-5bb52d1c3bb0\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6"
Oct 30 05:00:52.476764 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.476615    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79faca01-0058-4ef5-ab44-5bb52d1c3bb0-istiod-ca-cert\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6\" (UID: \"79faca01-0058-4ef5-ab44-5bb52d1c3bb0\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6"
Oct 30 05:00:52.476764 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.476654    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79faca01-0058-4ef5-ab44-5bb52d1c3bb0-workload-socket\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6\" (UID: \"79faca01-0058-4ef5-ab44-5bb52d1c3bb0\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6"
Oct 30 05:00:52.476764 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.476674    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79faca01-0058-4ef5-ab44-5bb52d1c3bb0-credential-socket\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6\" (UID: \"79faca01-0058-4ef5-ab44-5bb52d1c3bb0\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6"
Oct 30 05:00:52.476986 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.476811    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w95rq\" (UniqueName: \"kubernetes.io/projected/79faca01-0058-4ef5-ab44-5bb52d1c3bb0-kube-api-access-w95rq\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6\" (UID: \"79faca01-0058-4ef5-ab44-5bb52d1c3bb0\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6"
Oct 30 05:00:52.476986 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.476864    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79faca01-0058-4ef5-ab44-5bb52d1c3bb0-istio-podinfo\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6\" (UID: \"79faca01-0058-4ef5-ab44-5bb52d1c3bb0\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6"
Oct 30 05:00:52.476986 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.476939    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79faca01-0058-4ef5-ab44-5bb52d1c3bb0-workload-certs\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6\" (UID: \"79faca01-0058-4ef5-ab44-5bb52d1c3bb0\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6"
Oct 30 05:00:52.477203 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.477186    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79faca01-0058-4ef5-ab44-5bb52d1c3bb0-credential-socket\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6\" (UID: \"79faca01-0058-4ef5-ab44-5bb52d1c3bb0\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6"
Oct 30 05:00:52.477257 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.477236    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79faca01-0058-4ef5-ab44-5bb52d1c3bb0-istio-data\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6\" (UID: \"79faca01-0058-4ef5-ab44-5bb52d1c3bb0\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6"
Oct 30 05:00:52.477388 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.477366    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79faca01-0058-4ef5-ab44-5bb52d1c3bb0-istiod-ca-cert\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6\" (UID: \"79faca01-0058-4ef5-ab44-5bb52d1c3bb0\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6"
Oct 30 05:00:52.477442 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.477376    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79faca01-0058-4ef5-ab44-5bb52d1c3bb0-workload-socket\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6\" (UID: \"79faca01-0058-4ef5-ab44-5bb52d1c3bb0\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6"
Oct 30 05:00:52.478854 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.478823    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79faca01-0058-4ef5-ab44-5bb52d1c3bb0-istio-envoy\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6\" (UID: \"79faca01-0058-4ef5-ab44-5bb52d1c3bb0\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6"
Oct 30 05:00:52.479355 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.479338    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79faca01-0058-4ef5-ab44-5bb52d1c3bb0-istio-podinfo\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6\" (UID: \"79faca01-0058-4ef5-ab44-5bb52d1c3bb0\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6"
Oct 30 05:00:52.494511 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.494485    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79faca01-0058-4ef5-ab44-5bb52d1c3bb0-istio-token\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6\" (UID: \"79faca01-0058-4ef5-ab44-5bb52d1c3bb0\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6"
Oct 30 05:00:52.494596 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.494551    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w95rq\" (UniqueName: \"kubernetes.io/projected/79faca01-0058-4ef5-ab44-5bb52d1c3bb0-kube-api-access-w95rq\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6\" (UID: \"79faca01-0058-4ef5-ab44-5bb52d1c3bb0\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6"
Oct 30 05:00:52.496505 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.496486    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp"
Oct 30 05:00:52.612733 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.612701    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6"
Oct 30 05:00:52.804674 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.804642    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp"]
Oct 30 05:00:52.805119 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:00:52.805092    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode97ea58e_dd00_4d95_b178_806306ff635c.slice/crio-c2e1e3c14e92a27aa451be71126a81563060cdd7f053ba4a5622829ce78d9c14 WatchSource:0}: Error finding container c2e1e3c14e92a27aa451be71126a81563060cdd7f053ba4a5622829ce78d9c14: Status 404 returned error can't find the container with id c2e1e3c14e92a27aa451be71126a81563060cdd7f053ba4a5622829ce78d9c14
Oct 30 05:00:52.807782 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.807750    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:00:52.807886 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.807824    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:00:52.807886 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.807868    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:00:52.915376 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.915332    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6"]
Oct 30 05:00:52.916278 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:00:52.916254    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79faca01_0058_4ef5_ab44_5bb52d1c3bb0.slice/crio-dde0f1b0ca7236bc54e4bf21e31acbd27aa29ea937c9507645e8ee3dda57e55e WatchSource:0}: Error finding container dde0f1b0ca7236bc54e4bf21e31acbd27aa29ea937c9507645e8ee3dda57e55e: Status 404 returned error can't find the container with id dde0f1b0ca7236bc54e4bf21e31acbd27aa29ea937c9507645e8ee3dda57e55e
Oct 30 05:00:52.919046 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.919009    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:00:52.919129 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.919092    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:00:52.919167 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:52.919156    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:00:53.390794 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:53.390705    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:00:53.606086 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:53.606041    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6" event={"ID":"79faca01-0058-4ef5-ab44-5bb52d1c3bb0","Type":"ContainerStarted","Data":"d0d299da8d8a4c1b0efe4f1aabee42d6b5f15a6fec08fd012bed4af08c1081a2"}
Oct 30 05:00:53.606434 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:53.606401    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6" event={"ID":"79faca01-0058-4ef5-ab44-5bb52d1c3bb0","Type":"ContainerStarted","Data":"dde0f1b0ca7236bc54e4bf21e31acbd27aa29ea937c9507645e8ee3dda57e55e"}
Oct 30 05:00:53.611707 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:53.611673    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp" event={"ID":"e97ea58e-dd00-4d95-b178-806306ff635c","Type":"ContainerStarted","Data":"c1d05aa65baccbee37a898d2ed41aa95a28d777b6b1c02dbf43fb125c02ccadb"}
Oct 30 05:00:53.611991 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:53.611972    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp" event={"ID":"e97ea58e-dd00-4d95-b178-806306ff635c","Type":"ContainerStarted","Data":"c2e1e3c14e92a27aa451be71126a81563060cdd7f053ba4a5622829ce78d9c14"}
Oct 30 05:00:53.616228 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:53.616207    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6"
Oct 30 05:00:53.625055 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:53.625020    5898 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 05:00:53.633186 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:53.633146    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6" podUID="79faca01-0058-4ef5-ab44-5bb52d1c3bb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 05:00:53.700986 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:53.686542    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6" podStartSLOduration=1.686521854 podStartE2EDuration="1.686521854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:53.664744735 +0000 UTC m=+982.888564649" watchObservedRunningTime="2026-10-30 05:00:53.686521854 +0000 UTC m=+982.910341755"
Oct 30 05:00:53.732122 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:53.732055    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp" podStartSLOduration=1.7320353480000001 podStartE2EDuration="1.732035348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:53.716410818 +0000 UTC m=+982.940230722" watchObservedRunningTime="2026-10-30 05:00:53.732035348 +0000 UTC m=+982.955855249"
Oct 30 05:00:54.505331 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:54.505297    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp"
Oct 30 05:00:54.521045 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:54.521013    5898 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 05:00:54.521287 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:54.521251    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp" podUID="e97ea58e-dd00-4d95-b178-806306ff635c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 05:00:54.619859 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:54.619825    5898 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 05:00:54.624002 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:54.623963    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6" podUID="79faca01-0058-4ef5-ab44-5bb52d1c3bb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 05:00:55.437411 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:55.437367    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:00:55.507533 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:55.507498    5898 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 05:00:55.507812 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:55.507782    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp" podUID="e97ea58e-dd00-4d95-b178-806306ff635c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 05:00:55.621925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:55.621054    5898 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 05:00:55.621925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:55.621118    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6" podUID="79faca01-0058-4ef5-ab44-5bb52d1c3bb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 05:00:56.498699 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:56.498260    5898 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 05:00:56.498699 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:56.498319    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp" podUID="e97ea58e-dd00-4d95-b178-806306ff635c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 05:00:56.623291 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:56.623255    5898 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 05:00:56.624100 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:56.624021    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6" podUID="79faca01-0058-4ef5-ab44-5bb52d1c3bb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 05:00:56.843722 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:00:56.843686    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:00:57.256301 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:57.256195    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:00:57.507983 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:57.500265    5898 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 05:00:57.507983 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:57.500344    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp" podUID="e97ea58e-dd00-4d95-b178-806306ff635c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 05:00:57.619924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:57.614273    5898 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 05:00:57.619924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:57.614344    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6" podUID="79faca01-0058-4ef5-ab44-5bb52d1c3bb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 05:00:58.499309 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:58.499269    5898 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 05:00:58.499957 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:58.499924    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp" podUID="e97ea58e-dd00-4d95-b178-806306ff635c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 05:00:58.620313 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:58.620277    5898 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 05:00:58.620586 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:58.620555    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6" podUID="79faca01-0058-4ef5-ab44-5bb52d1c3bb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 05:00:59.235776 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:59.235743    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:00:59.497716 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:59.497637    5898 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 05:00:59.497716 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:59.497689    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp" podUID="e97ea58e-dd00-4d95-b178-806306ff635c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 05:00:59.894789 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:59.894757    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6"
Oct 30 05:00:59.895575 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:59.895277    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6"
Oct 30 05:00:59.896333 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:00:59.896307    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6"
Oct 30 05:01:00.238765 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:00.238684    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6"
Oct 30 05:01:00.251756 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:00.251730    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6"
Oct 30 05:01:00.501287 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:00.501208    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp"
Oct 30 05:01:00.501609 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:00.501588    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp"
Oct 30 05:01:00.502316 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:00.502295    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp"
Oct 30 05:01:00.689097 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:00.689074    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6"
Oct 30 05:01:01.253550 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:01.253513    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:01:01.626245 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:01.626212    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:01:01.848892 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:01.848855    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:01:03.238423 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:03.238388    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:01:04.585924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:04.585119    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247" podUID="1d865d3e-a6e9-407e-844e-cc75fc7f1354" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 05:01:04.726327 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:04.723047    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6" podUID="47ff272a-ac5c-4f59-bf7b-5c0692d09386" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:04.942092 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:04.941979    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5" podUID="ebf8f4a2-8ac6-48ec-bdba-3f6a99653264" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:05.273980 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:05.267997    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49" podUID="05eef4eb-f379-4dc7-a947-123bd85a4968" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 05:01:05.315112 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:05.315065    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6" podUID="b09804e3-8135-4f6d-ae34-c45d6da9728e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:05.446924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:05.446464    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58" podUID="c27f3b21-1447-4af9-8d99-c04fa9b23cda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 05:01:05.488927 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:05.485515    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx" podUID="d523008f-c740-4a41-89cb-e82e1ffa0094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 05:01:05.625926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:05.624988    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849" podUID="a9707f50-d5e1-409f-b8f3-c58629677aaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 05:01:05.840933 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:05.839855    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6" podUID="47ff272a-ac5c-4f59-bf7b-5c0692d09386" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Oct 30 05:01:05.840933 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:05.840258    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6"
Oct 30 05:01:05.945934 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:05.945805    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5" podUID="ebf8f4a2-8ac6-48ec-bdba-3f6a99653264" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:05.968930 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:05.946878    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5"
Oct 30 05:01:06.157046 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:06.156993    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9" podUID="1647f909-5580-4219-a2e1-30b4cd994379" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:06.291982 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:06.276026    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6"
Oct 30 05:01:06.291982 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:06.291672    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5"
Oct 30 05:01:06.921923 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:06.921869    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:01:06.994088 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:06.994058    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4"]
Oct 30 05:01:06.999267 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:06.999245    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4"
Oct 30 05:01:07.019285 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.019237    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4"]
Oct 30 05:01:07.055161 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.055115    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edceaaa4-f231-474e-85e1-b877ede202ef-istio-token\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4\" (UID: \"edceaaa4-f231-474e-85e1-b877ede202ef\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4"
Oct 30 05:01:07.055570 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.055516    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edceaaa4-f231-474e-85e1-b877ede202ef-credential-socket\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4\" (UID: \"edceaaa4-f231-474e-85e1-b877ede202ef\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4"
Oct 30 05:01:07.055724 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.055575    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edceaaa4-f231-474e-85e1-b877ede202ef-istiod-ca-cert\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4\" (UID: \"edceaaa4-f231-474e-85e1-b877ede202ef\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4"
Oct 30 05:01:07.055724 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.055632    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edceaaa4-f231-474e-85e1-b877ede202ef-workload-socket\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4\" (UID: \"edceaaa4-f231-474e-85e1-b877ede202ef\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4"
Oct 30 05:01:07.055724 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.055702    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edceaaa4-f231-474e-85e1-b877ede202ef-istio-podinfo\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4\" (UID: \"edceaaa4-f231-474e-85e1-b877ede202ef\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4"
Oct 30 05:01:07.055921 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.055757    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edceaaa4-f231-474e-85e1-b877ede202ef-istio-envoy\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4\" (UID: \"edceaaa4-f231-474e-85e1-b877ede202ef\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4"
Oct 30 05:01:07.055921 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.055780    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edceaaa4-f231-474e-85e1-b877ede202ef-istio-data\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4\" (UID: \"edceaaa4-f231-474e-85e1-b877ede202ef\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4"
Oct 30 05:01:07.055921 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.055834    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p74m\" (UniqueName: \"kubernetes.io/projected/edceaaa4-f231-474e-85e1-b877ede202ef-kube-api-access-8p74m\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4\" (UID: \"edceaaa4-f231-474e-85e1-b877ede202ef\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4"
Oct 30 05:01:07.056076 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.055955    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edceaaa4-f231-474e-85e1-b877ede202ef-workload-certs\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4\" (UID: \"edceaaa4-f231-474e-85e1-b877ede202ef\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4"
Oct 30 05:01:07.083015 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.082946    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q"]
Oct 30 05:01:07.087281 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.087259    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q"
Oct 30 05:01:07.109694 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.109663    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q"]
Oct 30 05:01:07.156868 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.156820    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edceaaa4-f231-474e-85e1-b877ede202ef-istio-envoy\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4\" (UID: \"edceaaa4-f231-474e-85e1-b877ede202ef\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4"
Oct 30 05:01:07.156868 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.156869    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edceaaa4-f231-474e-85e1-b877ede202ef-istio-data\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4\" (UID: \"edceaaa4-f231-474e-85e1-b877ede202ef\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4"
Oct 30 05:01:07.157141 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.156920    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e97be36-ccbe-43d0-8566-dfcca1d8f6a0-workload-socket\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q\" (UID: \"3e97be36-ccbe-43d0-8566-dfcca1d8f6a0\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q"
Oct 30 05:01:07.157141 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.157045    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p74m\" (UniqueName: \"kubernetes.io/projected/edceaaa4-f231-474e-85e1-b877ede202ef-kube-api-access-8p74m\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4\" (UID: \"edceaaa4-f231-474e-85e1-b877ede202ef\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4"
Oct 30 05:01:07.157262 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.157172    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e97be36-ccbe-43d0-8566-dfcca1d8f6a0-istiod-ca-cert\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q\" (UID: \"3e97be36-ccbe-43d0-8566-dfcca1d8f6a0\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q"
Oct 30 05:01:07.157262 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.157204    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e97be36-ccbe-43d0-8566-dfcca1d8f6a0-credential-socket\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q\" (UID: \"3e97be36-ccbe-43d0-8566-dfcca1d8f6a0\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q"
Oct 30 05:01:07.157377 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.157322    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edceaaa4-f231-474e-85e1-b877ede202ef-istio-data\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4\" (UID: \"edceaaa4-f231-474e-85e1-b877ede202ef\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4"
Oct 30 05:01:07.157377 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.157346    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edceaaa4-f231-474e-85e1-b877ede202ef-workload-certs\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4\" (UID: \"edceaaa4-f231-474e-85e1-b877ede202ef\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4"
Oct 30 05:01:07.157503 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.157445    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edceaaa4-f231-474e-85e1-b877ede202ef-istio-token\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4\" (UID: \"edceaaa4-f231-474e-85e1-b877ede202ef\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4"
Oct 30 05:01:07.157503 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.157490    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e97be36-ccbe-43d0-8566-dfcca1d8f6a0-istio-data\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q\" (UID: \"3e97be36-ccbe-43d0-8566-dfcca1d8f6a0\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q"
Oct 30 05:01:07.157616 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.157536    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edceaaa4-f231-474e-85e1-b877ede202ef-credential-socket\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4\" (UID: \"edceaaa4-f231-474e-85e1-b877ede202ef\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4"
Oct 30 05:01:07.157616 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.157577    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edceaaa4-f231-474e-85e1-b877ede202ef-istiod-ca-cert\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4\" (UID: \"edceaaa4-f231-474e-85e1-b877ede202ef\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4"
Oct 30 05:01:07.157616 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.157587    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edceaaa4-f231-474e-85e1-b877ede202ef-workload-certs\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4\" (UID: \"edceaaa4-f231-474e-85e1-b877ede202ef\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4"
Oct 30 05:01:07.157766 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.157654    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edceaaa4-f231-474e-85e1-b877ede202ef-workload-socket\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4\" (UID: \"edceaaa4-f231-474e-85e1-b877ede202ef\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4"
Oct 30 05:01:07.157766 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.157689    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e97be36-ccbe-43d0-8566-dfcca1d8f6a0-workload-certs\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q\" (UID: \"3e97be36-ccbe-43d0-8566-dfcca1d8f6a0\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q"
Oct 30 05:01:07.157766 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.157712    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e97be36-ccbe-43d0-8566-dfcca1d8f6a0-istio-envoy\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q\" (UID: \"3e97be36-ccbe-43d0-8566-dfcca1d8f6a0\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q"
Oct 30 05:01:07.157942 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.157805    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2qpm\" (UniqueName: \"kubernetes.io/projected/3e97be36-ccbe-43d0-8566-dfcca1d8f6a0-kube-api-access-t2qpm\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q\" (UID: \"3e97be36-ccbe-43d0-8566-dfcca1d8f6a0\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q"
Oct 30 05:01:07.157942 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.157893    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e97be36-ccbe-43d0-8566-dfcca1d8f6a0-istio-token\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q\" (UID: \"3e97be36-ccbe-43d0-8566-dfcca1d8f6a0\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q"
Oct 30 05:01:07.157942 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.157924    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edceaaa4-f231-474e-85e1-b877ede202ef-credential-socket\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4\" (UID: \"edceaaa4-f231-474e-85e1-b877ede202ef\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4"
Oct 30 05:01:07.158109 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.157949    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edceaaa4-f231-474e-85e1-b877ede202ef-istio-podinfo\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4\" (UID: \"edceaaa4-f231-474e-85e1-b877ede202ef\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4"
Oct 30 05:01:07.158109 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.157983    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edceaaa4-f231-474e-85e1-b877ede202ef-workload-socket\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4\" (UID: \"edceaaa4-f231-474e-85e1-b877ede202ef\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4"
Oct 30 05:01:07.158255 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.158230    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e97be36-ccbe-43d0-8566-dfcca1d8f6a0-istio-podinfo\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q\" (UID: \"3e97be36-ccbe-43d0-8566-dfcca1d8f6a0\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q"
Oct 30 05:01:07.158389 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.158365    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edceaaa4-f231-474e-85e1-b877ede202ef-istiod-ca-cert\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4\" (UID: \"edceaaa4-f231-474e-85e1-b877ede202ef\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4"
Oct 30 05:01:07.159420 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.159397    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edceaaa4-f231-474e-85e1-b877ede202ef-istio-envoy\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4\" (UID: \"edceaaa4-f231-474e-85e1-b877ede202ef\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4"
Oct 30 05:01:07.160280 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.160253    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edceaaa4-f231-474e-85e1-b877ede202ef-istio-podinfo\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4\" (UID: \"edceaaa4-f231-474e-85e1-b877ede202ef\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4"
Oct 30 05:01:07.176283 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.176228    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edceaaa4-f231-474e-85e1-b877ede202ef-istio-token\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4\" (UID: \"edceaaa4-f231-474e-85e1-b877ede202ef\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4"
Oct 30 05:01:07.176358 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.176279    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p74m\" (UniqueName: \"kubernetes.io/projected/edceaaa4-f231-474e-85e1-b877ede202ef-kube-api-access-8p74m\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4\" (UID: \"edceaaa4-f231-474e-85e1-b877ede202ef\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4"
Oct 30 05:01:07.259004 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.258978    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e97be36-ccbe-43d0-8566-dfcca1d8f6a0-credential-socket\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q\" (UID: \"3e97be36-ccbe-43d0-8566-dfcca1d8f6a0\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q"
Oct 30 05:01:07.259148 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.259074    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e97be36-ccbe-43d0-8566-dfcca1d8f6a0-istio-data\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q\" (UID: \"3e97be36-ccbe-43d0-8566-dfcca1d8f6a0\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q"
Oct 30 05:01:07.259148 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.259127    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e97be36-ccbe-43d0-8566-dfcca1d8f6a0-workload-certs\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q\" (UID: \"3e97be36-ccbe-43d0-8566-dfcca1d8f6a0\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q"
Oct 30 05:01:07.259148 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.259142    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e97be36-ccbe-43d0-8566-dfcca1d8f6a0-istio-envoy\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q\" (UID: \"3e97be36-ccbe-43d0-8566-dfcca1d8f6a0\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q"
Oct 30 05:01:07.259395 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.259172    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2qpm\" (UniqueName: \"kubernetes.io/projected/3e97be36-ccbe-43d0-8566-dfcca1d8f6a0-kube-api-access-t2qpm\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q\" (UID: \"3e97be36-ccbe-43d0-8566-dfcca1d8f6a0\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q"
Oct 30 05:01:07.259395 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.259200    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e97be36-ccbe-43d0-8566-dfcca1d8f6a0-istio-token\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q\" (UID: \"3e97be36-ccbe-43d0-8566-dfcca1d8f6a0\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q"
Oct 30 05:01:07.259395 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.259240    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e97be36-ccbe-43d0-8566-dfcca1d8f6a0-istio-podinfo\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q\" (UID: \"3e97be36-ccbe-43d0-8566-dfcca1d8f6a0\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q"
Oct 30 05:01:07.259395 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.259295    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e97be36-ccbe-43d0-8566-dfcca1d8f6a0-workload-socket\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q\" (UID: \"3e97be36-ccbe-43d0-8566-dfcca1d8f6a0\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q"
Oct 30 05:01:07.259395 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.259377    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e97be36-ccbe-43d0-8566-dfcca1d8f6a0-istiod-ca-cert\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q\" (UID: \"3e97be36-ccbe-43d0-8566-dfcca1d8f6a0\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q"
Oct 30 05:01:07.259665 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.259495    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e97be36-ccbe-43d0-8566-dfcca1d8f6a0-workload-certs\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q\" (UID: \"3e97be36-ccbe-43d0-8566-dfcca1d8f6a0\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q"
Oct 30 05:01:07.259665 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.259650    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e97be36-ccbe-43d0-8566-dfcca1d8f6a0-istio-data\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q\" (UID: \"3e97be36-ccbe-43d0-8566-dfcca1d8f6a0\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q"
Oct 30 05:01:07.260129 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.259761    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e97be36-ccbe-43d0-8566-dfcca1d8f6a0-credential-socket\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q\" (UID: \"3e97be36-ccbe-43d0-8566-dfcca1d8f6a0\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q"
Oct 30 05:01:07.260129 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.259956    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e97be36-ccbe-43d0-8566-dfcca1d8f6a0-workload-socket\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q\" (UID: \"3e97be36-ccbe-43d0-8566-dfcca1d8f6a0\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q"
Oct 30 05:01:07.260215 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.260180    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e97be36-ccbe-43d0-8566-dfcca1d8f6a0-istiod-ca-cert\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q\" (UID: \"3e97be36-ccbe-43d0-8566-dfcca1d8f6a0\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q"
Oct 30 05:01:07.261967 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.261942    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e97be36-ccbe-43d0-8566-dfcca1d8f6a0-istio-envoy\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q\" (UID: \"3e97be36-ccbe-43d0-8566-dfcca1d8f6a0\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q"
Oct 30 05:01:07.262182 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.262162    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e97be36-ccbe-43d0-8566-dfcca1d8f6a0-istio-podinfo\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q\" (UID: \"3e97be36-ccbe-43d0-8566-dfcca1d8f6a0\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q"
Oct 30 05:01:07.278393 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.278366    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e97be36-ccbe-43d0-8566-dfcca1d8f6a0-istio-token\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q\" (UID: \"3e97be36-ccbe-43d0-8566-dfcca1d8f6a0\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q"
Oct 30 05:01:07.278489 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.278448    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2qpm\" (UniqueName: \"kubernetes.io/projected/3e97be36-ccbe-43d0-8566-dfcca1d8f6a0-kube-api-access-t2qpm\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q\" (UID: \"3e97be36-ccbe-43d0-8566-dfcca1d8f6a0\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q"
Oct 30 05:01:07.317315 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.317292    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4"
Oct 30 05:01:07.407008 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.406972    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q"
Oct 30 05:01:07.562014 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.561982    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2"]
Oct 30 05:01:07.569774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.569316    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2"
Oct 30 05:01:07.597412 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.595498    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2"]
Oct 30 05:01:07.637233 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.636678    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4"]
Oct 30 05:01:07.639102 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:01:07.639075    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedceaaa4_f231_474e_85e1_b877ede202ef.slice/crio-a1ceeca5b289c5d539f7067f4767608761fc50a51be1ef48bbf8a1142742a538 WatchSource:0}: Error finding container a1ceeca5b289c5d539f7067f4767608761fc50a51be1ef48bbf8a1142742a538: Status 404 returned error can't find the container with id a1ceeca5b289c5d539f7067f4767608761fc50a51be1ef48bbf8a1142742a538
Oct 30 05:01:07.641933 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.641874    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:01:07.642026 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.641952    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:01:07.642026 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.641981    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:01:07.668397 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.668365    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/262d78f1-0e39-4673-a10d-fcf41d4522a4-istio-podinfo\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2\" (UID: \"262d78f1-0e39-4673-a10d-fcf41d4522a4\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2"
Oct 30 05:01:07.668520 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.668431    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5rtq\" (UniqueName: \"kubernetes.io/projected/262d78f1-0e39-4673-a10d-fcf41d4522a4-kube-api-access-j5rtq\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2\" (UID: \"262d78f1-0e39-4673-a10d-fcf41d4522a4\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2"
Oct 30 05:01:07.668567 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.668514    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/262d78f1-0e39-4673-a10d-fcf41d4522a4-istio-token\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2\" (UID: \"262d78f1-0e39-4673-a10d-fcf41d4522a4\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2"
Oct 30 05:01:07.668614 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.668570    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/262d78f1-0e39-4673-a10d-fcf41d4522a4-istiod-ca-cert\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2\" (UID: \"262d78f1-0e39-4673-a10d-fcf41d4522a4\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2"
Oct 30 05:01:07.668725 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.668710    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/262d78f1-0e39-4673-a10d-fcf41d4522a4-credential-socket\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2\" (UID: \"262d78f1-0e39-4673-a10d-fcf41d4522a4\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2"
Oct 30 05:01:07.668780 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.668733    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/262d78f1-0e39-4673-a10d-fcf41d4522a4-workload-certs\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2\" (UID: \"262d78f1-0e39-4673-a10d-fcf41d4522a4\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2"
Oct 30 05:01:07.668780 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.668756    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/262d78f1-0e39-4673-a10d-fcf41d4522a4-istio-envoy\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2\" (UID: \"262d78f1-0e39-4673-a10d-fcf41d4522a4\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2"
Oct 30 05:01:07.668886 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.668845    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/262d78f1-0e39-4673-a10d-fcf41d4522a4-workload-socket\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2\" (UID: \"262d78f1-0e39-4673-a10d-fcf41d4522a4\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2"
Oct 30 05:01:07.668978 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.668966    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/262d78f1-0e39-4673-a10d-fcf41d4522a4-istio-data\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2\" (UID: \"262d78f1-0e39-4673-a10d-fcf41d4522a4\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2"
Oct 30 05:01:07.718290 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.718264    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q"]
Oct 30 05:01:07.718674 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:01:07.718649    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e97be36_ccbe_43d0_8566_dfcca1d8f6a0.slice/crio-f1bc6127affc41189b98e4ad6dc32c70b9917546f04c49ccacd1fc402776e071 WatchSource:0}: Error finding container f1bc6127affc41189b98e4ad6dc32c70b9917546f04c49ccacd1fc402776e071: Status 404 returned error can't find the container with id f1bc6127affc41189b98e4ad6dc32c70b9917546f04c49ccacd1fc402776e071
Oct 30 05:01:07.721583 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.721551    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:01:07.721676 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.721639    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:01:07.721735 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.721687    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:01:07.769994 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.769956    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/262d78f1-0e39-4673-a10d-fcf41d4522a4-istio-podinfo\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2\" (UID: \"262d78f1-0e39-4673-a10d-fcf41d4522a4\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2"
Oct 30 05:01:07.770159 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.770042    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5rtq\" (UniqueName: \"kubernetes.io/projected/262d78f1-0e39-4673-a10d-fcf41d4522a4-kube-api-access-j5rtq\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2\" (UID: \"262d78f1-0e39-4673-a10d-fcf41d4522a4\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2"
Oct 30 05:01:07.770159 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.770070    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/262d78f1-0e39-4673-a10d-fcf41d4522a4-istio-token\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2\" (UID: \"262d78f1-0e39-4673-a10d-fcf41d4522a4\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2"
Oct 30 05:01:07.770159 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.770093    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/262d78f1-0e39-4673-a10d-fcf41d4522a4-istiod-ca-cert\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2\" (UID: \"262d78f1-0e39-4673-a10d-fcf41d4522a4\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2"
Oct 30 05:01:07.770334 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.770194    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/262d78f1-0e39-4673-a10d-fcf41d4522a4-credential-socket\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2\" (UID: \"262d78f1-0e39-4673-a10d-fcf41d4522a4\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2"
Oct 30 05:01:07.770334 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.770220    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/262d78f1-0e39-4673-a10d-fcf41d4522a4-workload-certs\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2\" (UID: \"262d78f1-0e39-4673-a10d-fcf41d4522a4\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2"
Oct 30 05:01:07.770334 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.770245    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/262d78f1-0e39-4673-a10d-fcf41d4522a4-istio-envoy\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2\" (UID: \"262d78f1-0e39-4673-a10d-fcf41d4522a4\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2"
Oct 30 05:01:07.770334 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.770279    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/262d78f1-0e39-4673-a10d-fcf41d4522a4-workload-socket\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2\" (UID: \"262d78f1-0e39-4673-a10d-fcf41d4522a4\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2"
Oct 30 05:01:07.770538 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.770379    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/262d78f1-0e39-4673-a10d-fcf41d4522a4-istio-data\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2\" (UID: \"262d78f1-0e39-4673-a10d-fcf41d4522a4\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2"
Oct 30 05:01:07.770851 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.770825    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/262d78f1-0e39-4673-a10d-fcf41d4522a4-istio-data\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2\" (UID: \"262d78f1-0e39-4673-a10d-fcf41d4522a4\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2"
Oct 30 05:01:07.771731 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.771705    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/262d78f1-0e39-4673-a10d-fcf41d4522a4-istiod-ca-cert\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2\" (UID: \"262d78f1-0e39-4673-a10d-fcf41d4522a4\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2"
Oct 30 05:01:07.772265 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.772246    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/262d78f1-0e39-4673-a10d-fcf41d4522a4-credential-socket\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2\" (UID: \"262d78f1-0e39-4673-a10d-fcf41d4522a4\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2"
Oct 30 05:01:07.772479 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.772452    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/262d78f1-0e39-4673-a10d-fcf41d4522a4-workload-certs\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2\" (UID: \"262d78f1-0e39-4673-a10d-fcf41d4522a4\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2"
Oct 30 05:01:07.772964 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.772942    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/262d78f1-0e39-4673-a10d-fcf41d4522a4-workload-socket\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2\" (UID: \"262d78f1-0e39-4673-a10d-fcf41d4522a4\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2"
Oct 30 05:01:07.773727 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.773684    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/262d78f1-0e39-4673-a10d-fcf41d4522a4-istio-podinfo\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2\" (UID: \"262d78f1-0e39-4673-a10d-fcf41d4522a4\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2"
Oct 30 05:01:07.774583 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.774565    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/262d78f1-0e39-4673-a10d-fcf41d4522a4-istio-envoy\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2\" (UID: \"262d78f1-0e39-4673-a10d-fcf41d4522a4\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2"
Oct 30 05:01:07.790890 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.790872    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5rtq\" (UniqueName: \"kubernetes.io/projected/262d78f1-0e39-4673-a10d-fcf41d4522a4-kube-api-access-j5rtq\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2\" (UID: \"262d78f1-0e39-4673-a10d-fcf41d4522a4\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2"
Oct 30 05:01:07.802879 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.802856    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/262d78f1-0e39-4673-a10d-fcf41d4522a4-istio-token\") pod \"e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2\" (UID: \"262d78f1-0e39-4673-a10d-fcf41d4522a4\") " pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2"
Oct 30 05:01:07.900582 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:07.900549    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2"
Oct 30 05:01:08.109314 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:08.109272    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q" event={"ID":"3e97be36-ccbe-43d0-8566-dfcca1d8f6a0","Type":"ContainerStarted","Data":"55a438e6b58925a6b419c43e1ca1c5832e3469205bf4c56454f28a1d30a84dad"}
Oct 30 05:01:08.109780 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:08.109347    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q" event={"ID":"3e97be36-ccbe-43d0-8566-dfcca1d8f6a0","Type":"ContainerStarted","Data":"f1bc6127affc41189b98e4ad6dc32c70b9917546f04c49ccacd1fc402776e071"}
Oct 30 05:01:08.111305 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:08.111214    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4" event={"ID":"edceaaa4-f231-474e-85e1-b877ede202ef","Type":"ContainerStarted","Data":"dea7f224f5dae13e8cdb887c420e975d8d10123af82b3bd4fe012a4926eca760"}
Oct 30 05:01:08.111305 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:08.111270    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4" event={"ID":"edceaaa4-f231-474e-85e1-b877ede202ef","Type":"ContainerStarted","Data":"a1ceeca5b289c5d539f7067f4767608761fc50a51be1ef48bbf8a1142742a538"}
Oct 30 05:01:08.200931 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:08.200177    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q" podStartSLOduration=1.200158861 podStartE2EDuration="1.200158861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:01:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:01:08.151599384 +0000 UTC m=+997.375419289" watchObservedRunningTime="2026-10-30 05:01:08.200158861 +0000 UTC m=+997.423978765"
Oct 30 05:01:08.249667 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:08.247627    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:01:08.249667 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:08.247727    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:01:08.249667 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:08.247772    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:01:08.259108 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:08.258279    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4" podStartSLOduration=2.2582598799999998 podStartE2EDuration="2.25825988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:01:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:01:08.209603286 +0000 UTC m=+997.433423206" watchObservedRunningTime="2026-10-30 05:01:08.25825988 +0000 UTC m=+997.482079782"
Oct 30 05:01:08.259108 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:08.258729    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2"]
Oct 30 05:01:08.336147 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:08.325976    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4"
Oct 30 05:01:08.336788 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:08.336757    5898 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 05:01:08.336915 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:08.336815    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4" podUID="edceaaa4-f231-474e-85e1-b877ede202ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 05:01:08.419845 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:08.419773    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q"
Oct 30 05:01:08.420755 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:08.420466    5898 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 05:01:08.420755 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:08.420519    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q" podUID="3e97be36-ccbe-43d0-8566-dfcca1d8f6a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 05:01:09.063278 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:09.063248    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:01:09.146405 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:09.146358    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2" event={"ID":"262d78f1-0e39-4673-a10d-fcf41d4522a4","Type":"ContainerStarted","Data":"76abacc1cdf1739384e41a9e93c5b59c0eb1bc7b5b35dd46ac44165a254a2624"}
Oct 30 05:01:09.146976 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:09.146952    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2" event={"ID":"262d78f1-0e39-4673-a10d-fcf41d4522a4","Type":"ContainerStarted","Data":"46a8247ba423e21e194ad22dbd5803af21404ce8dfef5a75dd9aa5e6c1fcb2d4"}
Oct 30 05:01:09.206925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:09.205778    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2" podStartSLOduration=2.2057564530000002 podStartE2EDuration="2.205756453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:01:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:01:09.193178063 +0000 UTC m=+998.416997962" watchObservedRunningTime="2026-10-30 05:01:09.205756453 +0000 UTC m=+998.429576355"
Oct 30 05:01:09.320986 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:09.319025    5898 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 05:01:09.320986 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:09.319104    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4" podUID="edceaaa4-f231-474e-85e1-b877ede202ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 05:01:09.410426 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:09.410391    5898 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 05:01:09.410896 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:09.410862    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q" podUID="3e97be36-ccbe-43d0-8566-dfcca1d8f6a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 05:01:09.501925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:09.494407    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:01:09.915925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:09.915212    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2"
Oct 30 05:01:09.915925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:09.915352    5898 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 05:01:09.915925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:09.915400    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2" podUID="262d78f1-0e39-4673-a10d-fcf41d4522a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 05:01:10.320985 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:10.320276    5898 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 05:01:10.320985 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:10.320336    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4" podUID="edceaaa4-f231-474e-85e1-b877ede202ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 05:01:10.429925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:10.429054    5898 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 05:01:10.429925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:10.429119    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q" podUID="3e97be36-ccbe-43d0-8566-dfcca1d8f6a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 05:01:10.902044 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:10.902005    5898 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 05:01:10.902252 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:10.902069    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2" podUID="262d78f1-0e39-4673-a10d-fcf41d4522a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 05:01:11.332981 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:11.329005    5898 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 05:01:11.332981 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:11.329063    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4" podUID="edceaaa4-f231-474e-85e1-b877ede202ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 05:01:11.416927 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:11.408301    5898 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 05:01:11.416927 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:11.408362    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q" podUID="3e97be36-ccbe-43d0-8566-dfcca1d8f6a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 05:01:11.881927 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:11.881876    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:01:11.902921 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:11.902289    5898 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 05:01:11.902921 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:11.902349    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2" podUID="262d78f1-0e39-4673-a10d-fcf41d4522a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 05:01:12.324301 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:12.324269    5898 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 05:01:12.324574 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:12.324547    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4" podUID="edceaaa4-f231-474e-85e1-b877ede202ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 05:01:12.409287 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:12.409247    5898 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 05:01:12.409866 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:12.409836    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q" podUID="3e97be36-ccbe-43d0-8566-dfcca1d8f6a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 05:01:12.907064 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:12.907022    5898 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 05:01:12.907280 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:12.907092    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2" podUID="262d78f1-0e39-4673-a10d-fcf41d4522a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 05:01:13.328927 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:13.328812    5898 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 05:01:13.328927 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:13.328874    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4" podUID="edceaaa4-f231-474e-85e1-b877ede202ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 05:01:13.412329 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:13.412294    5898 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 05:01:13.412943 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:13.412892    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q" podUID="3e97be36-ccbe-43d0-8566-dfcca1d8f6a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 05:01:13.906374 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:13.906334    5898 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 05:01:13.906691 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:13.906652    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2" podUID="262d78f1-0e39-4673-a10d-fcf41d4522a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 05:01:14.318198 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:14.318098    5898 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 05:01:14.318198 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:14.318173    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4" podUID="edceaaa4-f231-474e-85e1-b877ede202ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 05:01:14.321061 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:14.321032    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:01:14.408320 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:14.408283    5898 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 05:01:14.408589 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:14.408559    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q" podUID="3e97be36-ccbe-43d0-8566-dfcca1d8f6a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 05:01:14.900871 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:14.900836    5898 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 05:01:14.901406 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:14.900888    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2" podUID="262d78f1-0e39-4673-a10d-fcf41d4522a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 05:01:15.464005 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:15.463949    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4"
Oct 30 05:01:15.465823 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:15.465796    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4"
Oct 30 05:01:15.466516 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:15.466494    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4"
Oct 30 05:01:15.763982 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:15.763118    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q"
Oct 30 05:01:15.763982 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:15.763792    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q"
Oct 30 05:01:15.764820 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:15.764795    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q"
Oct 30 05:01:15.906049 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:15.906006    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2"
Oct 30 05:01:15.906540 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:15.906501    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2"
Oct 30 05:01:15.907727 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:15.907708    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2"
Oct 30 05:01:16.653199 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:16.653162    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:01:18.536784 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:18.536750    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:01:20.245511 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:20.245480    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:01:21.684917 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:21.684872    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm"]
Oct 30 05:01:21.689016 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:21.688990    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm"
Oct 30 05:01:21.711953 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:21.711929    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp"]
Oct 30 05:01:21.715931 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:21.715909    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp"
Oct 30 05:01:21.724819 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:21.724794    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm"]
Oct 30 05:01:21.745863 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:21.745832    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp"]
Oct 30 05:01:21.826580 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:21.826516    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e371f321-3fd2-4ce5-96a5-625388d49ba0-credential-socket\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm\" (UID: \"e371f321-3fd2-4ce5-96a5-625388d49ba0\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm"
Oct 30 05:01:21.826580 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:21.826577    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa97913c-19e1-432c-9bd2-b1ce3278f7a5-istiod-ca-cert\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp\" (UID: \"fa97913c-19e1-432c-9bd2-b1ce3278f7a5\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp"
Oct 30 05:01:21.826785 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:21.826599    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e371f321-3fd2-4ce5-96a5-625388d49ba0-istio-token\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm\" (UID: \"e371f321-3fd2-4ce5-96a5-625388d49ba0\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm"
Oct 30 05:01:21.826852 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:21.826827    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa97913c-19e1-432c-9bd2-b1ce3278f7a5-workload-socket\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp\" (UID: \"fa97913c-19e1-432c-9bd2-b1ce3278f7a5\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp"
Oct 30 05:01:21.826937 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:21.826896    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7448q\" (UniqueName: \"kubernetes.io/projected/fa97913c-19e1-432c-9bd2-b1ce3278f7a5-kube-api-access-7448q\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp\" (UID: \"fa97913c-19e1-432c-9bd2-b1ce3278f7a5\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp"
Oct 30 05:01:21.827073 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:21.826973    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa97913c-19e1-432c-9bd2-b1ce3278f7a5-credential-socket\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp\" (UID: \"fa97913c-19e1-432c-9bd2-b1ce3278f7a5\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp"
Oct 30 05:01:21.827073 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:21.827034    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e371f321-3fd2-4ce5-96a5-625388d49ba0-istiod-ca-cert\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm\" (UID: \"e371f321-3fd2-4ce5-96a5-625388d49ba0\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm"
Oct 30 05:01:21.827168 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:21.827141    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2ndm\" (UniqueName: \"kubernetes.io/projected/e371f321-3fd2-4ce5-96a5-625388d49ba0-kube-api-access-q2ndm\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm\" (UID: \"e371f321-3fd2-4ce5-96a5-625388d49ba0\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm"
Oct 30 05:01:21.827218 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:21.827202    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e371f321-3fd2-4ce5-96a5-625388d49ba0-workload-socket\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm\" (UID: \"e371f321-3fd2-4ce5-96a5-625388d49ba0\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm"
Oct 30 05:01:21.827255 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:21.827241    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa97913c-19e1-432c-9bd2-b1ce3278f7a5-istio-data\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp\" (UID: \"fa97913c-19e1-432c-9bd2-b1ce3278f7a5\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp"
Oct 30 05:01:21.827310 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:21.827271    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e371f321-3fd2-4ce5-96a5-625388d49ba0-istio-data\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm\" (UID: \"e371f321-3fd2-4ce5-96a5-625388d49ba0\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm"
Oct 30 05:01:21.827344 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:21.827320    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa97913c-19e1-432c-9bd2-b1ce3278f7a5-istio-podinfo\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp\" (UID: \"fa97913c-19e1-432c-9bd2-b1ce3278f7a5\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp"
Oct 30 05:01:21.827420 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:21.827408    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e371f321-3fd2-4ce5-96a5-625388d49ba0-istio-podinfo\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm\" (UID: \"e371f321-3fd2-4ce5-96a5-625388d49ba0\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm"
Oct 30 05:01:21.827455 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:21.827430    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e371f321-3fd2-4ce5-96a5-625388d49ba0-workload-certs\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm\" (UID: \"e371f321-3fd2-4ce5-96a5-625388d49ba0\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm"
Oct 30 05:01:21.827555 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:21.827539    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa97913c-19e1-432c-9bd2-b1ce3278f7a5-workload-certs\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp\" (UID: \"fa97913c-19e1-432c-9bd2-b1ce3278f7a5\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp"
Oct 30 05:01:21.827600 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:21.827583    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa97913c-19e1-432c-9bd2-b1ce3278f7a5-istio-envoy\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp\" (UID: \"fa97913c-19e1-432c-9bd2-b1ce3278f7a5\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp"
Oct 30 05:01:21.827698 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:21.827679    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e371f321-3fd2-4ce5-96a5-625388d49ba0-istio-envoy\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm\" (UID: \"e371f321-3fd2-4ce5-96a5-625388d49ba0\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm"
Oct 30 05:01:21.827740 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:21.827726    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa97913c-19e1-432c-9bd2-b1ce3278f7a5-istio-token\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp\" (UID: \"fa97913c-19e1-432c-9bd2-b1ce3278f7a5\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp"
Oct 30 05:01:21.928841 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:21.928804    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa97913c-19e1-432c-9bd2-b1ce3278f7a5-workload-socket\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp\" (UID: \"fa97913c-19e1-432c-9bd2-b1ce3278f7a5\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp"
Oct 30 05:01:21.929227 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:21.929199    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7448q\" (UniqueName: \"kubernetes.io/projected/fa97913c-19e1-432c-9bd2-b1ce3278f7a5-kube-api-access-7448q\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp\" (UID: \"fa97913c-19e1-432c-9bd2-b1ce3278f7a5\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp"
Oct 30 05:01:21.929367 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:21.929338    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa97913c-19e1-432c-9bd2-b1ce3278f7a5-workload-socket\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp\" (UID: \"fa97913c-19e1-432c-9bd2-b1ce3278f7a5\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp"
Oct 30 05:01:21.929548 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:21.929365    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa97913c-19e1-432c-9bd2-b1ce3278f7a5-credential-socket\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp\" (UID: \"fa97913c-19e1-432c-9bd2-b1ce3278f7a5\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp"
Oct 30 05:01:21.929685 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:21.929657    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa97913c-19e1-432c-9bd2-b1ce3278f7a5-credential-socket\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp\" (UID: \"fa97913c-19e1-432c-9bd2-b1ce3278f7a5\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp"
Oct 30 05:01:21.929795 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:21.929693    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e371f321-3fd2-4ce5-96a5-625388d49ba0-istiod-ca-cert\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm\" (UID: \"e371f321-3fd2-4ce5-96a5-625388d49ba0\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm"
Oct 30 05:01:21.929795 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:21.929762    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2ndm\" (UniqueName: \"kubernetes.io/projected/e371f321-3fd2-4ce5-96a5-625388d49ba0-kube-api-access-q2ndm\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm\" (UID: \"e371f321-3fd2-4ce5-96a5-625388d49ba0\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm"
Oct 30 05:01:21.929942 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:21.929807    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e371f321-3fd2-4ce5-96a5-625388d49ba0-workload-socket\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm\" (UID: \"e371f321-3fd2-4ce5-96a5-625388d49ba0\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm"
Oct 30 05:01:21.929942 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:21.929840    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa97913c-19e1-432c-9bd2-b1ce3278f7a5-istio-data\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp\" (UID: \"fa97913c-19e1-432c-9bd2-b1ce3278f7a5\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp"
Oct 30 05:01:21.929942 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:21.929865    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e371f321-3fd2-4ce5-96a5-625388d49ba0-istio-data\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm\" (UID: \"e371f321-3fd2-4ce5-96a5-625388d49ba0\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm"
Oct 30 05:01:21.929942 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:21.929888    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa97913c-19e1-432c-9bd2-b1ce3278f7a5-istio-podinfo\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp\" (UID: \"fa97913c-19e1-432c-9bd2-b1ce3278f7a5\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp"
Oct 30 05:01:21.930166 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:21.929958    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e371f321-3fd2-4ce5-96a5-625388d49ba0-istio-podinfo\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm\" (UID: \"e371f321-3fd2-4ce5-96a5-625388d49ba0\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm"
Oct 30 05:01:21.930166 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:21.929986    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e371f321-3fd2-4ce5-96a5-625388d49ba0-workload-certs\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm\" (UID: \"e371f321-3fd2-4ce5-96a5-625388d49ba0\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm"
Oct 30 05:01:21.930166 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:21.930060    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa97913c-19e1-432c-9bd2-b1ce3278f7a5-workload-certs\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp\" (UID: \"fa97913c-19e1-432c-9bd2-b1ce3278f7a5\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp"
Oct 30 05:01:21.930166 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:21.930100    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa97913c-19e1-432c-9bd2-b1ce3278f7a5-istio-envoy\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp\" (UID: \"fa97913c-19e1-432c-9bd2-b1ce3278f7a5\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp"
Oct 30 05:01:21.930166 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:21.930146    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e371f321-3fd2-4ce5-96a5-625388d49ba0-istio-envoy\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm\" (UID: \"e371f321-3fd2-4ce5-96a5-625388d49ba0\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm"
Oct 30 05:01:21.930701 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:21.930176    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa97913c-19e1-432c-9bd2-b1ce3278f7a5-istio-token\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp\" (UID: \"fa97913c-19e1-432c-9bd2-b1ce3278f7a5\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp"
Oct 30 05:01:21.930701 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:21.930222    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e371f321-3fd2-4ce5-96a5-625388d49ba0-credential-socket\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm\" (UID: \"e371f321-3fd2-4ce5-96a5-625388d49ba0\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm"
Oct 30 05:01:21.930701 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:21.930249    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa97913c-19e1-432c-9bd2-b1ce3278f7a5-istiod-ca-cert\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp\" (UID: \"fa97913c-19e1-432c-9bd2-b1ce3278f7a5\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp"
Oct 30 05:01:21.930701 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:21.930278    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e371f321-3fd2-4ce5-96a5-625388d49ba0-istio-token\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm\" (UID: \"e371f321-3fd2-4ce5-96a5-625388d49ba0\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm"
Oct 30 05:01:21.930701 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:21.930605    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e371f321-3fd2-4ce5-96a5-625388d49ba0-istiod-ca-cert\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm\" (UID: \"e371f321-3fd2-4ce5-96a5-625388d49ba0\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm"
Oct 30 05:01:21.930992 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:21.930925    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e371f321-3fd2-4ce5-96a5-625388d49ba0-workload-socket\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm\" (UID: \"e371f321-3fd2-4ce5-96a5-625388d49ba0\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm"
Oct 30 05:01:21.931595 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:21.931219    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa97913c-19e1-432c-9bd2-b1ce3278f7a5-istio-data\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp\" (UID: \"fa97913c-19e1-432c-9bd2-b1ce3278f7a5\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp"
Oct 30 05:01:21.931595 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:21.931245    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e371f321-3fd2-4ce5-96a5-625388d49ba0-workload-certs\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm\" (UID: \"e371f321-3fd2-4ce5-96a5-625388d49ba0\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm"
Oct 30 05:01:21.931595 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:21.931276    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa97913c-19e1-432c-9bd2-b1ce3278f7a5-workload-certs\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp\" (UID: \"fa97913c-19e1-432c-9bd2-b1ce3278f7a5\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp"
Oct 30 05:01:21.931595 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:21.931353    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e371f321-3fd2-4ce5-96a5-625388d49ba0-istio-data\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm\" (UID: \"e371f321-3fd2-4ce5-96a5-625388d49ba0\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm"
Oct 30 05:01:21.931595 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:21.931382    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e371f321-3fd2-4ce5-96a5-625388d49ba0-credential-socket\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm\" (UID: \"e371f321-3fd2-4ce5-96a5-625388d49ba0\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm"
Oct 30 05:01:21.931972 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:21.931947    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa97913c-19e1-432c-9bd2-b1ce3278f7a5-istiod-ca-cert\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp\" (UID: \"fa97913c-19e1-432c-9bd2-b1ce3278f7a5\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp"
Oct 30 05:01:21.934150 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:21.934123    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa97913c-19e1-432c-9bd2-b1ce3278f7a5-istio-podinfo\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp\" (UID: \"fa97913c-19e1-432c-9bd2-b1ce3278f7a5\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp"
Oct 30 05:01:21.935616 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:21.935550    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e371f321-3fd2-4ce5-96a5-625388d49ba0-istio-envoy\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm\" (UID: \"e371f321-3fd2-4ce5-96a5-625388d49ba0\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm"
Oct 30 05:01:21.935740 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:21.935714    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa97913c-19e1-432c-9bd2-b1ce3278f7a5-istio-envoy\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp\" (UID: \"fa97913c-19e1-432c-9bd2-b1ce3278f7a5\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp"
Oct 30 05:01:21.935890 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:21.935777    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e371f321-3fd2-4ce5-96a5-625388d49ba0-istio-podinfo\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm\" (UID: \"e371f321-3fd2-4ce5-96a5-625388d49ba0\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm"
Oct 30 05:01:21.948919 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:21.948853    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2ndm\" (UniqueName: \"kubernetes.io/projected/e371f321-3fd2-4ce5-96a5-625388d49ba0-kube-api-access-q2ndm\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm\" (UID: \"e371f321-3fd2-4ce5-96a5-625388d49ba0\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm"
Oct 30 05:01:21.949039 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:21.948994    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e371f321-3fd2-4ce5-96a5-625388d49ba0-istio-token\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm\" (UID: \"e371f321-3fd2-4ce5-96a5-625388d49ba0\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm"
Oct 30 05:01:21.949664 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:21.949642    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7448q\" (UniqueName: \"kubernetes.io/projected/fa97913c-19e1-432c-9bd2-b1ce3278f7a5-kube-api-access-7448q\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp\" (UID: \"fa97913c-19e1-432c-9bd2-b1ce3278f7a5\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp"
Oct 30 05:01:21.958260 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:21.958233    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa97913c-19e1-432c-9bd2-b1ce3278f7a5-istio-token\") pod \"e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp\" (UID: \"fa97913c-19e1-432c-9bd2-b1ce3278f7a5\") " pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp"
Oct 30 05:01:22.004049 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:22.004015    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm"
Oct 30 05:01:22.035847 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:22.035607    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp"
Oct 30 05:01:22.122256 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:22.118739    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9"]
Oct 30 05:01:22.134859 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:22.132019    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9"
Oct 30 05:01:22.145673 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:22.145646    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9"]
Oct 30 05:01:22.277972 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:22.240226    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62d9b95-5024-49d1-88f4-048477578a45-credential-socket\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9\" (UID: \"d62d9b95-5024-49d1-88f4-048477578a45\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9"
Oct 30 05:01:22.277972 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:22.240376    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d62d9b95-5024-49d1-88f4-048477578a45-istio-podinfo\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9\" (UID: \"d62d9b95-5024-49d1-88f4-048477578a45\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9"
Oct 30 05:01:22.277972 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:22.240453    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d62d9b95-5024-49d1-88f4-048477578a45-workload-certs\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9\" (UID: \"d62d9b95-5024-49d1-88f4-048477578a45\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9"
Oct 30 05:01:22.277972 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:22.240476    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d62d9b95-5024-49d1-88f4-048477578a45-istio-envoy\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9\" (UID: \"d62d9b95-5024-49d1-88f4-048477578a45\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9"
Oct 30 05:01:22.277972 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:22.240507    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d62d9b95-5024-49d1-88f4-048477578a45-istiod-ca-cert\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9\" (UID: \"d62d9b95-5024-49d1-88f4-048477578a45\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9"
Oct 30 05:01:22.277972 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:22.240551    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d62d9b95-5024-49d1-88f4-048477578a45-istio-data\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9\" (UID: \"d62d9b95-5024-49d1-88f4-048477578a45\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9"
Oct 30 05:01:22.277972 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:22.240638    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz29j\" (UniqueName: \"kubernetes.io/projected/d62d9b95-5024-49d1-88f4-048477578a45-kube-api-access-zz29j\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9\" (UID: \"d62d9b95-5024-49d1-88f4-048477578a45\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9"
Oct 30 05:01:22.277972 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:22.240668    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d62d9b95-5024-49d1-88f4-048477578a45-istio-token\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9\" (UID: \"d62d9b95-5024-49d1-88f4-048477578a45\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9"
Oct 30 05:01:22.277972 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:22.240743    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62d9b95-5024-49d1-88f4-048477578a45-workload-socket\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9\" (UID: \"d62d9b95-5024-49d1-88f4-048477578a45\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9"
Oct 30 05:01:22.388767 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:22.354219    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62d9b95-5024-49d1-88f4-048477578a45-credential-socket\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9\" (UID: \"d62d9b95-5024-49d1-88f4-048477578a45\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9"
Oct 30 05:01:22.388767 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:22.354387    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d62d9b95-5024-49d1-88f4-048477578a45-istio-podinfo\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9\" (UID: \"d62d9b95-5024-49d1-88f4-048477578a45\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9"
Oct 30 05:01:22.388767 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:22.354467    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d62d9b95-5024-49d1-88f4-048477578a45-workload-certs\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9\" (UID: \"d62d9b95-5024-49d1-88f4-048477578a45\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9"
Oct 30 05:01:22.388767 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:22.354493    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d62d9b95-5024-49d1-88f4-048477578a45-istio-envoy\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9\" (UID: \"d62d9b95-5024-49d1-88f4-048477578a45\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9"
Oct 30 05:01:22.388767 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:22.354528    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d62d9b95-5024-49d1-88f4-048477578a45-istiod-ca-cert\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9\" (UID: \"d62d9b95-5024-49d1-88f4-048477578a45\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9"
Oct 30 05:01:22.388767 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:22.354577    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d62d9b95-5024-49d1-88f4-048477578a45-istio-data\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9\" (UID: \"d62d9b95-5024-49d1-88f4-048477578a45\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9"
Oct 30 05:01:22.388767 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:22.354673    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz29j\" (UniqueName: \"kubernetes.io/projected/d62d9b95-5024-49d1-88f4-048477578a45-kube-api-access-zz29j\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9\" (UID: \"d62d9b95-5024-49d1-88f4-048477578a45\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9"
Oct 30 05:01:22.388767 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:22.354706    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d62d9b95-5024-49d1-88f4-048477578a45-istio-token\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9\" (UID: \"d62d9b95-5024-49d1-88f4-048477578a45\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9"
Oct 30 05:01:22.388767 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:22.354782    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62d9b95-5024-49d1-88f4-048477578a45-workload-socket\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9\" (UID: \"d62d9b95-5024-49d1-88f4-048477578a45\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9"
Oct 30 05:01:22.388767 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:22.381685    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62d9b95-5024-49d1-88f4-048477578a45-workload-socket\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9\" (UID: \"d62d9b95-5024-49d1-88f4-048477578a45\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9"
Oct 30 05:01:22.388767 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:22.382406    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d62d9b95-5024-49d1-88f4-048477578a45-istiod-ca-cert\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9\" (UID: \"d62d9b95-5024-49d1-88f4-048477578a45\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9"
Oct 30 05:01:22.388767 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:22.382674    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d62d9b95-5024-49d1-88f4-048477578a45-istio-data\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9\" (UID: \"d62d9b95-5024-49d1-88f4-048477578a45\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9"
Oct 30 05:01:22.388767 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:22.387187    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d62d9b95-5024-49d1-88f4-048477578a45-workload-certs\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9\" (UID: \"d62d9b95-5024-49d1-88f4-048477578a45\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9"
Oct 30 05:01:22.426924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:22.391186    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62d9b95-5024-49d1-88f4-048477578a45-credential-socket\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9\" (UID: \"d62d9b95-5024-49d1-88f4-048477578a45\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9"
Oct 30 05:01:22.843925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:22.833688    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz29j\" (UniqueName: \"kubernetes.io/projected/d62d9b95-5024-49d1-88f4-048477578a45-kube-api-access-zz29j\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9\" (UID: \"d62d9b95-5024-49d1-88f4-048477578a45\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9"
Oct 30 05:01:23.014855 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:23.014746    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d62d9b95-5024-49d1-88f4-048477578a45-istio-token\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9\" (UID: \"d62d9b95-5024-49d1-88f4-048477578a45\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9"
Oct 30 05:01:23.021493 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:23.021454    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d62d9b95-5024-49d1-88f4-048477578a45-istio-envoy\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9\" (UID: \"d62d9b95-5024-49d1-88f4-048477578a45\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9"
Oct 30 05:01:23.051932 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:23.051795    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d62d9b95-5024-49d1-88f4-048477578a45-istio-podinfo\") pod \"e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9\" (UID: \"d62d9b95-5024-49d1-88f4-048477578a45\") " pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9"
Oct 30 05:01:23.151659 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:23.151091    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g" podUID="70294a85-1050-4632-b3bf-c3e44a4c0b41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:23.206112 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:23.204992    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9" podUID="1647f909-5580-4219-a2e1-30b4cd994379" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:23.241039 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:23.239244    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9"
Oct 30 05:01:23.282251 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:23.259399    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:01:23.282251 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:23.259486    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:01:23.282251 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:23.259532    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:01:23.299813 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:23.290967    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6" podUID="47ff272a-ac5c-4f59-bf7b-5c0692d09386" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:23.299813 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:23.293298    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49" podUID="05eef4eb-f379-4dc7-a947-123bd85a4968" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:23.299813 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:23.293359    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb" podUID="6f16bd6a-a7f0-49bb-844f-57b777bde2ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:23.299813 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:23.294296    5898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247" podUID="1d865d3e-a6e9-407e-844e-cc75fc7f1354" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": context deadline exceeded"
Oct 30 05:01:23.299813 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:23.294768    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6" podUID="b09804e3-8135-4f6d-ae34-c45d6da9728e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 05:01:23.332592 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:23.330989    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5" podUID="ebf8f4a2-8ac6-48ec-bdba-3f6a99653264" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 05:01:23.332592 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:23.331282    5898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh" podUID="38428a53-04c7-400a-9c78-52407e982478" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": context deadline exceeded"
Oct 30 05:01:23.336028 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:23.335944    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm"]
Oct 30 05:01:23.413925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:23.404480    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp"]
Oct 30 05:01:23.522287 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:23.521014    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58" podUID="c27f3b21-1447-4af9-8d99-c04fa9b23cda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:23.524068 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:23.522529    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx" podUID="d523008f-c740-4a41-89cb-e82e1ffa0094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 05:01:23.524068 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:23.523130    5898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx" podUID="d523008f-c740-4a41-89cb-e82e1ffa0094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:23.649335 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:23.641761    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247" podUID="1d865d3e-a6e9-407e-844e-cc75fc7f1354" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:23.649335 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:23.646125    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:01:23.649335 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:23.646535    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:01:23.649335 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:23.647120    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:01:23.649335 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:23.647849    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849" podUID="a9707f50-d5e1-409f-b8f3-c58629677aaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 05:01:23.649335 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:23.648829    5898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb" podUID="6f16bd6a-a7f0-49bb-844f-57b777bde2ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:23.655926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:23.650451    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh" podUID="38428a53-04c7-400a-9c78-52407e982478" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:23.753325 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:23.730365    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:01:23.813624 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:23.813574    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp" event={"ID":"fa97913c-19e1-432c-9bd2-b1ce3278f7a5","Type":"ContainerStarted","Data":"e844e208aed5dfaa602fb132cef5599212b3fd9cf2811aa0946e61f1f1b555a7"}
Oct 30 05:01:23.840243 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:23.836345    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm" event={"ID":"e371f321-3fd2-4ce5-96a5-625388d49ba0","Type":"ContainerStarted","Data":"55915a2c3e965e185896feaadbb1592373f201a5340efde0540ff4a49f2828e9"}
Oct 30 05:01:24.155942 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:24.153341    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g" podUID="70294a85-1050-4632-b3bf-c3e44a4c0b41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:24.155942 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:24.154048    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g"
Oct 30 05:01:24.272925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:24.242102    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9" podUID="1647f909-5580-4219-a2e1-30b4cd994379" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 05:01:24.272925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:24.242852    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9"
Oct 30 05:01:24.335922 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:24.298961    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6" podUID="b09804e3-8135-4f6d-ae34-c45d6da9728e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:24.335922 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:24.299334    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6" podUID="47ff272a-ac5c-4f59-bf7b-5c0692d09386" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Oct 30 05:01:24.362925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:24.338361    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb" podUID="6f16bd6a-a7f0-49bb-844f-57b777bde2ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:24.385426 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:24.343446    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6"
Oct 30 05:01:24.385426 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:24.383424    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6"
Oct 30 05:01:24.385426 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:24.383466    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb"
Oct 30 05:01:24.385426 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:24.351060    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5" podUID="ebf8f4a2-8ac6-48ec-bdba-3f6a99653264" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 05:01:24.389036 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:24.388690    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5"
Oct 30 05:01:24.596924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:24.579050    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58" podUID="c27f3b21-1447-4af9-8d99-c04fa9b23cda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:24.687208 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:24.594071    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx" podUID="d523008f-c740-4a41-89cb-e82e1ffa0094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:24.801515 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:24.299604    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49" podUID="05eef4eb-f379-4dc7-a947-123bd85a4968" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 05:01:24.801515 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:24.646585    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247" podUID="1d865d3e-a6e9-407e-844e-cc75fc7f1354" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 05:01:24.801515 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:24.687764    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58"
Oct 30 05:01:24.801515 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:24.777081    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx"
Oct 30 05:01:24.801515 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:24.777126    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49"
Oct 30 05:01:24.801515 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:24.777162    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247"
Oct 30 05:01:24.801515 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:24.748675    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849" podUID="a9707f50-d5e1-409f-b8f3-c58629677aaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:24.801515 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:24.778643    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849"
Oct 30 05:01:24.801515 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:24.750520    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh" podUID="38428a53-04c7-400a-9c78-52407e982478" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 30 05:01:24.801515 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:24.780769    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh"
Oct 30 05:01:24.878646 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:24.871484    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6"
Oct 30 05:01:25.168280 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:25.164058    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g" podUID="70294a85-1050-4632-b3bf-c3e44a4c0b41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:25.232866 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:01:25.232548    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd62d9b95_5024_49d1_88f4_048477578a45.slice/crio-52b62a664410fb2af5b4d5c6ab67644912afb37acde83a0d4a8164f1038a5f6a WatchSource:0}: Error finding container 52b62a664410fb2af5b4d5c6ab67644912afb37acde83a0d4a8164f1038a5f6a: Status 404 returned error can't find the container with id 52b62a664410fb2af5b4d5c6ab67644912afb37acde83a0d4a8164f1038a5f6a
Oct 30 05:01:25.269926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:25.269272    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9" podUID="1647f909-5580-4219-a2e1-30b4cd994379" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 05:01:25.286291 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:25.286246    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:01:25.286583 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:25.286562    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:01:25.286777 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:25.286739    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:01:25.423536 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:25.393689    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5" podUID="ebf8f4a2-8ac6-48ec-bdba-3f6a99653264" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:25.423536 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:25.394096    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb" podUID="6f16bd6a-a7f0-49bb-844f-57b777bde2ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 05:01:25.688196 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:25.688056    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58" podUID="c27f3b21-1447-4af9-8d99-c04fa9b23cda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:25.776404 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:25.776360    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49" podUID="05eef4eb-f379-4dc7-a947-123bd85a4968" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:25.777036 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:25.776642    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx" podUID="d523008f-c740-4a41-89cb-e82e1ffa0094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:25.785279 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:25.777501    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247" podUID="1d865d3e-a6e9-407e-844e-cc75fc7f1354" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:25.807696 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:25.779866    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849" podUID="a9707f50-d5e1-409f-b8f3-c58629677aaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:25.808153 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:25.784964    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh" podUID="38428a53-04c7-400a-9c78-52407e982478" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:25.889182 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:25.889142    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6" podUID="b09804e3-8135-4f6d-ae34-c45d6da9728e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 05:01:25.914223 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:01:25.914191    5898 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.022s"
Oct 30 05:01:25.934191 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:25.934147    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm" event={"ID":"e371f321-3fd2-4ce5-96a5-625388d49ba0","Type":"ContainerStarted","Data":"75087f6cc857392c37e2f5080f6aa5bcfad125578c6dd4fbec6c06267e518d2b"}
Oct 30 05:01:25.934488 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:25.934471    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6"
Oct 30 05:01:25.990990 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:25.985987    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6" podUID="47ff272a-ac5c-4f59-bf7b-5c0692d09386" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:26.036924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:26.018729    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm"
Oct 30 05:01:26.037307 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:26.037277    5898 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 05:01:26.037499 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:26.037472    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm" podUID="e371f321-3fd2-4ce5-96a5-625388d49ba0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 05:01:26.166933 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:26.164609    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g" podUID="70294a85-1050-4632-b3bf-c3e44a4c0b41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:26.229717 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:26.229675    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp" event={"ID":"fa97913c-19e1-432c-9bd2-b1ce3278f7a5","Type":"ContainerStarted","Data":"0d431a2552cdab88b3e21fa66e0a7a7321fbc890b500e7a693d010be9e80209a"}
Oct 30 05:01:26.255923 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:26.246188    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9" event={"ID":"d62d9b95-5024-49d1-88f4-048477578a45","Type":"ContainerStarted","Data":"52b62a664410fb2af5b4d5c6ab67644912afb37acde83a0d4a8164f1038a5f6a"}
Oct 30 05:01:26.302940 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:26.281351    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9" podUID="1647f909-5580-4219-a2e1-30b4cd994379" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:26.393990 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:26.393947    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5" podUID="ebf8f4a2-8ac6-48ec-bdba-3f6a99653264" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:26.394774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:26.394439    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb" podUID="6f16bd6a-a7f0-49bb-844f-57b777bde2ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:26.597248 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:26.597216    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9"]
Oct 30 05:01:26.961574 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:26.959171    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx" podUID="d523008f-c740-4a41-89cb-e82e1ffa0094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:26.961574 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:26.960099    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh" podUID="38428a53-04c7-400a-9c78-52407e982478" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:26.961574 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:26.960939    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58" podUID="c27f3b21-1447-4af9-8d99-c04fa9b23cda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 05:01:26.961574 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:26.961431    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49" podUID="05eef4eb-f379-4dc7-a947-123bd85a4968" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 05:01:26.961929 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:26.961839    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247" podUID="1d865d3e-a6e9-407e-844e-cc75fc7f1354" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 05:01:27.165940 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:26.962424    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849" podUID="a9707f50-d5e1-409f-b8f3-c58629677aaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:27.172920 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:27.171050    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g" podUID="70294a85-1050-4632-b3bf-c3e44a4c0b41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 05:01:27.241980 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:27.222974    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6" podUID="47ff272a-ac5c-4f59-bf7b-5c0692d09386" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Oct 30 05:01:27.241980 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:27.238047    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6" podUID="b09804e3-8135-4f6d-ae34-c45d6da9728e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 05:01:27.242590 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:27.242114    5898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58" podUID="c27f3b21-1447-4af9-8d99-c04fa9b23cda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:27.242752 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:27.242725    5898 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 05:01:27.242971 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:27.242950    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm" podUID="e371f321-3fd2-4ce5-96a5-625388d49ba0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 05:01:27.276957 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:27.276921    5898 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 05:01:27.277230 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:27.277205    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp" podUID="fa97913c-19e1-432c-9bd2-b1ce3278f7a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 05:01:27.329573 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:27.329531    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9" podUID="1647f909-5580-4219-a2e1-30b4cd994379" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:27.462228 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:27.462198    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp"
Oct 30 05:01:27.462535 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:27.462513    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9" event={"ID":"d62d9b95-5024-49d1-88f4-048477578a45","Type":"ContainerStarted","Data":"42bf18f22b3abc85d7391e970ec6c296716b394e068e8ec33a998416b51b9f5b"}
Oct 30 05:01:27.462693 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:27.462680    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5"
Oct 30 05:01:27.462825 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:27.462816    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6"
Oct 30 05:01:27.462951 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:27.462941    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6"
Oct 30 05:01:27.463062 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:27.463054    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb"
Oct 30 05:01:27.730941 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:01:27.726501    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:01:27.754423 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:27.754397    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49"
Oct 30 05:01:27.755756 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:27.755713    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm" podStartSLOduration=6.755699006 podStartE2EDuration="6.755699006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:01:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:01:25.393452637 +0000 UTC m=+1014.617272538" watchObservedRunningTime="2026-10-30 05:01:27.755699006 +0000 UTC m=+1016.979518952"
Oct 30 05:01:27.884836 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:27.884711    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849"
Oct 30 05:01:27.903344 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:27.903302    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247"
Oct 30 05:01:27.908013 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:27.907986    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9"
Oct 30 05:01:27.922932 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:27.922888    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx"
Oct 30 05:01:27.962482 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:27.962446    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58" podUID="c27f3b21-1447-4af9-8d99-c04fa9b23cda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:27.963023 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:27.962996    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh" podUID="38428a53-04c7-400a-9c78-52407e982478" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 30 05:01:28.014319 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:28.014150    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58"
Oct 30 05:01:28.037690 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:28.037654    5898 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 05:01:28.063210 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:28.063165    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm" podUID="e371f321-3fd2-4ce5-96a5-625388d49ba0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 05:01:28.063442 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:28.042947    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6"
Oct 30 05:01:28.063561 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:28.042305    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp" podStartSLOduration=7.042282304 podStartE2EDuration="7.042282304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:01:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:01:26.326456602 +0000 UTC m=+1015.550276523" watchObservedRunningTime="2026-10-30 05:01:28.042282304 +0000 UTC m=+1017.266102209"
Oct 30 05:01:28.067939 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:28.038195    5898 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 05:01:28.067939 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:28.067040    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp" podUID="fa97913c-19e1-432c-9bd2-b1ce3278f7a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 05:01:28.075417 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:28.075390    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:01:28.112335 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:28.112276    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9" podStartSLOduration=6.093190559 podStartE2EDuration="6.093190559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:01:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:01:27.431419054 +0000 UTC m=+1016.655239064" watchObservedRunningTime="2026-10-30 05:01:28.093190559 +0000 UTC m=+1017.317010461"
Oct 30 05:01:28.138956 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:28.128453    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6"
Oct 30 05:01:28.138956 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:28.128659    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh"
Oct 30 05:01:28.174789 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:28.174750    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g" podUID="70294a85-1050-4632-b3bf-c3e44a4c0b41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:28.340453 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:28.340418    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9"
Oct 30 05:01:28.445354 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:28.423082    5898 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 05:01:28.445354 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:28.423143    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9" podUID="d62d9b95-5024-49d1-88f4-048477578a45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 05:01:28.468928 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:28.456987    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5" podUID="ebf8f4a2-8ac6-48ec-bdba-3f6a99653264" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:28.723003 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:28.714811    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:01:29.018392 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:29.018299    5898 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 05:01:29.018700 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:29.018668    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm" podUID="e371f321-3fd2-4ce5-96a5-625388d49ba0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 05:01:29.064004 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:29.063930    5898 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 05:01:29.064226 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:29.064006    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp" podUID="fa97913c-19e1-432c-9bd2-b1ce3278f7a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 05:01:29.135926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:29.124644    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g"
Oct 30 05:01:29.222122 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:29.222076    5898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5" podUID="ebf8f4a2-8ac6-48ec-bdba-3f6a99653264" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:29.245600 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:29.245561    5898 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 05:01:29.245885 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:29.245853    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9" podUID="d62d9b95-5024-49d1-88f4-048477578a45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 05:01:29.343050 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:29.343008    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9" podUID="1647f909-5580-4219-a2e1-30b4cd994379" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:30.006922 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:30.006315    5898 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 05:01:30.006922 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:30.006385    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm" podUID="e371f321-3fd2-4ce5-96a5-625388d49ba0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 05:01:30.038930 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:30.038498    5898 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 05:01:30.038930 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:30.038560    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp" podUID="fa97913c-19e1-432c-9bd2-b1ce3278f7a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 05:01:30.248922 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:30.244402    5898 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 05:01:30.248922 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:30.244460    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9" podUID="d62d9b95-5024-49d1-88f4-048477578a45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 05:01:30.864056 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:30.851503    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:01:31.016843 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:31.016807    5898 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 05:01:31.017134 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:31.017100    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm" podUID="e371f321-3fd2-4ce5-96a5-625388d49ba0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 05:01:31.039976 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:31.039820    5898 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 05:01:31.039976 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:31.039895    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp" podUID="fa97913c-19e1-432c-9bd2-b1ce3278f7a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 05:01:31.268993 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:31.253693    5898 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 05:01:31.268993 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:31.253758    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9" podUID="d62d9b95-5024-49d1-88f4-048477578a45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 05:01:32.010347 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:32.010304    5898 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 05:01:32.011038 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:32.010998    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm" podUID="e371f321-3fd2-4ce5-96a5-625388d49ba0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 05:01:32.037335 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:32.037297    5898 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 05:01:32.037607 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:32.037579    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp" podUID="fa97913c-19e1-432c-9bd2-b1ce3278f7a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 05:01:32.250925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:32.248277    5898 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 05:01:32.250925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:32.248339    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9" podUID="d62d9b95-5024-49d1-88f4-048477578a45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 05:01:32.872829 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:32.872798    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:01:33.008480 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:33.008438    5898 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 05:01:33.008685 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:33.008502    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm" podUID="e371f321-3fd2-4ce5-96a5-625388d49ba0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 05:01:33.037416 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:33.037381    5898 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 05:01:33.038038 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:33.038002    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp" podUID="fa97913c-19e1-432c-9bd2-b1ce3278f7a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 05:01:33.249830 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:33.249739    5898 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 05:01:33.250193 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:33.250162    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9" podUID="d62d9b95-5024-49d1-88f4-048477578a45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 05:01:34.008764 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:34.008300    5898 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 05:01:34.008764 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:34.008374    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm" podUID="e371f321-3fd2-4ce5-96a5-625388d49ba0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 05:01:34.036993 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:34.036955    5898 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 05:01:34.037167 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:34.037019    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp" podUID="fa97913c-19e1-432c-9bd2-b1ce3278f7a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 05:01:34.245300 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:34.245256    5898 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 05:01:34.245805 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:34.245324    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9" podUID="d62d9b95-5024-49d1-88f4-048477578a45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 05:01:34.754967 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:34.754893    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:01:35.011178 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:35.011093    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm"
Oct 30 05:01:35.012637 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:35.012614    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm"
Oct 30 05:01:35.013459 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:35.013439    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm"
Oct 30 05:01:35.352216 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:35.352009    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp"
Oct 30 05:01:35.353138 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:35.353039    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp"
Oct 30 05:01:35.353974 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:35.353954    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp"
Oct 30 05:01:35.635465 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:35.635377    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9"
Oct 30 05:01:35.635945 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:35.635875    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9"
Oct 30 05:01:35.636826 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:35.636807    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9"
Oct 30 05:01:36.735211 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:36.735176    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:01:38.441277 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:38.441245    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:01:38.549063 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:38.549025    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:01:40.165153 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:40.165123    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:01:41.959287 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:41.959253    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:01:43.639269 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:43.639228    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:01:44.276546 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:44.276490    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:01:45.372488 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:45.372452    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:01:47.049705 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:47.049671    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:01:47.150135 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:47.150088    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:01:48.113162 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:48.113120    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:01:48.738148 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:48.738107    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:01:50.813053 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:50.813006    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh" podUID="38428a53-04c7-400a-9c78-52407e982478" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 30 05:01:50.814316 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:50.813645    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247" podUID="1d865d3e-a6e9-407e-844e-cc75fc7f1354" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 05:01:50.814810 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:50.813674    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849" podUID="a9707f50-d5e1-409f-b8f3-c58629677aaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 05:01:50.845236 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:50.845195    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6" podUID="b09804e3-8135-4f6d-ae34-c45d6da9728e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 05:01:50.936074 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:50.936024    5898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6" podUID="b09804e3-8135-4f6d-ae34-c45d6da9728e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": context deadline exceeded"
Oct 30 05:01:50.959925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:50.936628    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx" podUID="d523008f-c740-4a41-89cb-e82e1ffa0094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 05:01:51.007263 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:51.007222    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49" podUID="05eef4eb-f379-4dc7-a947-123bd85a4968" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 05:01:51.058032 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:51.057984    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6" podUID="47ff272a-ac5c-4f59-bf7b-5c0692d09386" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:51.132058 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:51.131993    5898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49" podUID="05eef4eb-f379-4dc7-a947-123bd85a4968" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:51.172094 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:51.172032    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g" podUID="70294a85-1050-4632-b3bf-c3e44a4c0b41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:51.248096 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:51.248040    5898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6" podUID="47ff272a-ac5c-4f59-bf7b-5c0692d09386" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:51.333651 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:51.333498    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9" podUID="1647f909-5580-4219-a2e1-30b4cd994379" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:51.405319 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:51.405208    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb" podUID="6f16bd6a-a7f0-49bb-844f-57b777bde2ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 05:01:51.405954 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:51.405921    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5" podUID="ebf8f4a2-8ac6-48ec-bdba-3f6a99653264" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 05:01:51.702095 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:51.701993    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58" podUID="c27f3b21-1447-4af9-8d99-c04fa9b23cda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 05:01:51.817028 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:51.816982    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh" podUID="38428a53-04c7-400a-9c78-52407e982478" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 30 05:01:51.818281 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:51.818257    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh"
Oct 30 05:01:51.820026 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:51.819999    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849" podUID="a9707f50-d5e1-409f-b8f3-c58629677aaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 05:01:51.820439 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:51.820414    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247" podUID="1d865d3e-a6e9-407e-844e-cc75fc7f1354" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:51.821249 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:51.820986    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849"
Oct 30 05:01:51.838039 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:51.838013    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247"
Oct 30 05:01:51.852017 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:51.851980    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6" podUID="b09804e3-8135-4f6d-ae34-c45d6da9728e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:51.852588 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:51.852566    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6"
Oct 30 05:01:51.971985 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:51.971753    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx" podUID="d523008f-c740-4a41-89cb-e82e1ffa0094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:51.997927 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:51.972784    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx"
Oct 30 05:01:52.177074 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:52.177013    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g" podUID="70294a85-1050-4632-b3bf-c3e44a4c0b41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:52.527565 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:52.212951    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6" podUID="47ff272a-ac5c-4f59-bf7b-5c0692d09386" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:52.528251 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:52.528221    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g"
Oct 30 05:01:52.528472 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:52.528458    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6"
Oct 30 05:01:52.528575 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:52.467477    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5" podUID="ebf8f4a2-8ac6-48ec-bdba-3f6a99653264" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:52.528755 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:52.227281    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49" podUID="05eef4eb-f379-4dc7-a947-123bd85a4968" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:52.529033 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:52.360076    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9" podUID="1647f909-5580-4219-a2e1-30b4cd994379" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:52.529322 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:52.467519    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb" podUID="6f16bd6a-a7f0-49bb-844f-57b777bde2ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 05:01:52.566343 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:52.566312    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5"
Oct 30 05:01:52.566642 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:52.566625    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49"
Oct 30 05:01:52.566819 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:52.566806    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9"
Oct 30 05:01:52.566999 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:52.566984    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb"
Oct 30 05:01:52.732921 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:52.732396    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58" podUID="c27f3b21-1447-4af9-8d99-c04fa9b23cda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:52.888720 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:52.885684    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6" podUID="b09804e3-8135-4f6d-ae34-c45d6da9728e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:52.888720 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:52.886799    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh" podUID="38428a53-04c7-400a-9c78-52407e982478" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 30 05:01:52.888720 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:52.888685    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58"
Oct 30 05:01:52.889565 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:52.889535    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849" podUID="a9707f50-d5e1-409f-b8f3-c58629677aaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 05:01:52.937928 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:52.934607    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247" podUID="1d865d3e-a6e9-407e-844e-cc75fc7f1354" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 05:01:53.029925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:52.986788    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx" podUID="d523008f-c740-4a41-89cb-e82e1ffa0094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 05:01:53.181008 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:53.164774    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd"]
Oct 30 05:01:53.214400 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:53.214371    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd"
Oct 30 05:01:53.239920 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:53.234342    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd"]
Oct 30 05:01:53.292195 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:53.292160    5898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247" podUID="1d865d3e-a6e9-407e-844e-cc75fc7f1354" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:53.348996 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:53.348953    5898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh" podUID="38428a53-04c7-400a-9c78-52407e982478" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:53.402924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:53.402365    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ced35944-e749-49e0-97ac-806b824c601c-istio-data\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd\" (UID: \"ced35944-e749-49e0-97ac-806b824c601c\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd"
Oct 30 05:01:53.402924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:53.402469    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5scr4\" (UniqueName: \"kubernetes.io/projected/ced35944-e749-49e0-97ac-806b824c601c-kube-api-access-5scr4\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd\" (UID: \"ced35944-e749-49e0-97ac-806b824c601c\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd"
Oct 30 05:01:53.402924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:53.402588    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ced35944-e749-49e0-97ac-806b824c601c-credential-socket\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd\" (UID: \"ced35944-e749-49e0-97ac-806b824c601c\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd"
Oct 30 05:01:53.402924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:53.402613    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ced35944-e749-49e0-97ac-806b824c601c-istio-podinfo\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd\" (UID: \"ced35944-e749-49e0-97ac-806b824c601c\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd"
Oct 30 05:01:53.402924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:53.402661    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ced35944-e749-49e0-97ac-806b824c601c-workload-certs\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd\" (UID: \"ced35944-e749-49e0-97ac-806b824c601c\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd"
Oct 30 05:01:53.402924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:53.402688    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ced35944-e749-49e0-97ac-806b824c601c-istio-token\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd\" (UID: \"ced35944-e749-49e0-97ac-806b824c601c\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd"
Oct 30 05:01:53.402924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:53.402776    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ced35944-e749-49e0-97ac-806b824c601c-workload-socket\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd\" (UID: \"ced35944-e749-49e0-97ac-806b824c601c\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd"
Oct 30 05:01:53.402924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:53.402842    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ced35944-e749-49e0-97ac-806b824c601c-istiod-ca-cert\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd\" (UID: \"ced35944-e749-49e0-97ac-806b824c601c\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd"
Oct 30 05:01:53.402924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:53.402922    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ced35944-e749-49e0-97ac-806b824c601c-istio-envoy\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd\" (UID: \"ced35944-e749-49e0-97ac-806b824c601c\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd"
Oct 30 05:01:53.489747 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:53.489656    5898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb" podUID="6f16bd6a-a7f0-49bb-844f-57b777bde2ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:53.490406 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:53.490377    5898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx" podUID="d523008f-c740-4a41-89cb-e82e1ffa0094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:53.511075 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:53.508626    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ced35944-e749-49e0-97ac-806b824c601c-credential-socket\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd\" (UID: \"ced35944-e749-49e0-97ac-806b824c601c\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd"
Oct 30 05:01:53.511075 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:53.508669    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ced35944-e749-49e0-97ac-806b824c601c-istio-podinfo\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd\" (UID: \"ced35944-e749-49e0-97ac-806b824c601c\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd"
Oct 30 05:01:53.511075 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:53.508713    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ced35944-e749-49e0-97ac-806b824c601c-workload-certs\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd\" (UID: \"ced35944-e749-49e0-97ac-806b824c601c\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd"
Oct 30 05:01:53.511075 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:53.508740    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ced35944-e749-49e0-97ac-806b824c601c-istio-token\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd\" (UID: \"ced35944-e749-49e0-97ac-806b824c601c\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd"
Oct 30 05:01:53.511075 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:53.508814    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ced35944-e749-49e0-97ac-806b824c601c-workload-socket\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd\" (UID: \"ced35944-e749-49e0-97ac-806b824c601c\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd"
Oct 30 05:01:53.511075 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:53.508870    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ced35944-e749-49e0-97ac-806b824c601c-istiod-ca-cert\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd\" (UID: \"ced35944-e749-49e0-97ac-806b824c601c\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd"
Oct 30 05:01:53.511075 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:53.508939    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ced35944-e749-49e0-97ac-806b824c601c-istio-envoy\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd\" (UID: \"ced35944-e749-49e0-97ac-806b824c601c\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd"
Oct 30 05:01:53.511075 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:53.508992    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ced35944-e749-49e0-97ac-806b824c601c-istio-data\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd\" (UID: \"ced35944-e749-49e0-97ac-806b824c601c\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd"
Oct 30 05:01:53.511075 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:53.509061    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5scr4\" (UniqueName: \"kubernetes.io/projected/ced35944-e749-49e0-97ac-806b824c601c-kube-api-access-5scr4\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd\" (UID: \"ced35944-e749-49e0-97ac-806b824c601c\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd"
Oct 30 05:01:53.511075 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:53.509733    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ced35944-e749-49e0-97ac-806b824c601c-credential-socket\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd\" (UID: \"ced35944-e749-49e0-97ac-806b824c601c\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd"
Oct 30 05:01:53.511075 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:53.510365    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ced35944-e749-49e0-97ac-806b824c601c-workload-socket\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd\" (UID: \"ced35944-e749-49e0-97ac-806b824c601c\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd"
Oct 30 05:01:53.511075 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:53.510568    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ced35944-e749-49e0-97ac-806b824c601c-workload-certs\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd\" (UID: \"ced35944-e749-49e0-97ac-806b824c601c\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd"
Oct 30 05:01:53.512525 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:53.512501    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ced35944-e749-49e0-97ac-806b824c601c-istiod-ca-cert\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd\" (UID: \"ced35944-e749-49e0-97ac-806b824c601c\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd"
Oct 30 05:01:53.512914 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:53.512880    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ced35944-e749-49e0-97ac-806b824c601c-istio-data\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd\" (UID: \"ced35944-e749-49e0-97ac-806b824c601c\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd"
Oct 30 05:01:53.527771 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:53.527743    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g" podUID="70294a85-1050-4632-b3bf-c3e44a4c0b41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 05:01:53.528100 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:53.527947    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6" podUID="47ff272a-ac5c-4f59-bf7b-5c0692d09386" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Oct 30 05:01:53.528758 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:53.528734    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5" podUID="ebf8f4a2-8ac6-48ec-bdba-3f6a99653264" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:53.532913 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:53.529155    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49" podUID="05eef4eb-f379-4dc7-a947-123bd85a4968" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:53.534402 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:53.534378    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb" podUID="6f16bd6a-a7f0-49bb-844f-57b777bde2ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:53.542169 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:53.539971    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9" podUID="1647f909-5580-4219-a2e1-30b4cd994379" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 05:01:53.557498 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:53.557468    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ced35944-e749-49e0-97ac-806b824c601c-istio-token\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd\" (UID: \"ced35944-e749-49e0-97ac-806b824c601c\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd"
Oct 30 05:01:53.604403 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:53.604373    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb"
Oct 30 05:01:53.615977 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:53.615812    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ced35944-e749-49e0-97ac-806b824c601c-istio-podinfo\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd\" (UID: \"ced35944-e749-49e0-97ac-806b824c601c\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd"
Oct 30 05:01:53.686579 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:53.686408    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5scr4\" (UniqueName: \"kubernetes.io/projected/ced35944-e749-49e0-97ac-806b824c601c-kube-api-access-5scr4\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd\" (UID: \"ced35944-e749-49e0-97ac-806b824c601c\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd"
Oct 30 05:01:53.709710 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:53.701523    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ced35944-e749-49e0-97ac-806b824c601c-istio-envoy\") pod \"e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd\" (UID: \"ced35944-e749-49e0-97ac-806b824c601c\") " pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd"
Oct 30 05:01:53.721190 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:53.721153    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd"
Oct 30 05:01:53.783468 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:53.782256    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g"
Oct 30 05:01:53.888050 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:53.888004    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh" podUID="38428a53-04c7-400a-9c78-52407e982478" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:53.888628 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:53.888597    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6" podUID="b09804e3-8135-4f6d-ae34-c45d6da9728e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 05:01:53.898922 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:53.892594    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58" podUID="c27f3b21-1447-4af9-8d99-c04fa9b23cda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:53.929715 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:53.929668    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849" podUID="a9707f50-d5e1-409f-b8f3-c58629677aaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:53.936588 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:53.935524    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247" podUID="1d865d3e-a6e9-407e-844e-cc75fc7f1354" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:54.007503 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:54.006350    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247"
Oct 30 05:01:54.066509 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:54.063146    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh"
Oct 30 05:01:54.131013 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:54.130974    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx" podUID="d523008f-c740-4a41-89cb-e82e1ffa0094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:54.472009 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:54.471223    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49"
Oct 30 05:01:54.540694 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:54.540650    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5" podUID="ebf8f4a2-8ac6-48ec-bdba-3f6a99653264" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:54.541467 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:54.541431    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6" podUID="47ff272a-ac5c-4f59-bf7b-5c0692d09386" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Oct 30 05:01:54.545503 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:54.544859    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9" podUID="1647f909-5580-4219-a2e1-30b4cd994379" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:54.898524 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:54.898475    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6" podUID="b09804e3-8135-4f6d-ae34-c45d6da9728e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:54.900126 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:54.900094    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58" podUID="c27f3b21-1447-4af9-8d99-c04fa9b23cda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:54.977333 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:54.939360    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849" podUID="a9707f50-d5e1-409f-b8f3-c58629677aaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:55.258279 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:55.246014    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx" podUID="d523008f-c740-4a41-89cb-e82e1ffa0094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 05:01:55.527588 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:55.522471    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49" podUID="05eef4eb-f379-4dc7-a947-123bd85a4968" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:55.544986 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:55.543298    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6" podUID="47ff272a-ac5c-4f59-bf7b-5c0692d09386" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Oct 30 05:01:55.544986 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:55.543668    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5" podUID="ebf8f4a2-8ac6-48ec-bdba-3f6a99653264" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:55.545252 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:55.545006    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9" podUID="1647f909-5580-4219-a2e1-30b4cd994379" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:55.591446 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:55.591397    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd"]
Oct 30 05:01:55.894012 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:55.730668    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb" podUID="6f16bd6a-a7f0-49bb-844f-57b777bde2ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:55.895276 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:01:55.895238    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podced35944_e749_49e0_97ac_806b824c601c.slice/crio-433ff2fe513d5690e04c04e7c293e03bc8710e79900c859bf921d51fdc5f8427 WatchSource:0}: Error finding container 433ff2fe513d5690e04c04e7c293e03bc8710e79900c859bf921d51fdc5f8427: Status 404 returned error can't find the container with id 433ff2fe513d5690e04c04e7c293e03bc8710e79900c859bf921d51fdc5f8427
Oct 30 05:01:55.917932 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:55.903814    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:01:55.917932 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:55.910046    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:01:55.917932 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:55.910160    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:01:55.917932 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:55.910814    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh" podUID="38428a53-04c7-400a-9c78-52407e982478" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:55.926736 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:55.923497    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6" podUID="b09804e3-8135-4f6d-ae34-c45d6da9728e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 05:01:55.978452 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:55.978413    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58" podUID="c27f3b21-1447-4af9-8d99-c04fa9b23cda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 05:01:56.016468 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:56.014710    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849" podUID="a9707f50-d5e1-409f-b8f3-c58629677aaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:56.248489 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:56.248449    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx" podUID="d523008f-c740-4a41-89cb-e82e1ffa0094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 05:01:56.282012 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:56.281974    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx"
Oct 30 05:01:56.318575 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:56.318547    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6"
Oct 30 05:01:56.461174 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:56.454004    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58"
Oct 30 05:01:56.542920 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:56.526032    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49" podUID="05eef4eb-f379-4dc7-a947-123bd85a4968" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:56.542920 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:56.526709    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49"
Oct 30 05:01:56.597916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:56.592042    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9" podUID="1647f909-5580-4219-a2e1-30b4cd994379" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:01:56.597916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:56.592244    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6" podUID="47ff272a-ac5c-4f59-bf7b-5c0692d09386" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Oct 30 05:01:56.597916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:56.592503    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5" podUID="ebf8f4a2-8ac6-48ec-bdba-3f6a99653264" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 05:01:56.659351 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:56.657460    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9"
Oct 30 05:01:56.666625 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:56.666583    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49"
Oct 30 05:01:56.666936 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:56.666914    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6"
Oct 30 05:01:56.678991 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:56.678958    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849"
Oct 30 05:01:56.679199 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:56.679178    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5"
Oct 30 05:01:56.828533 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:01:56.828496    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:01:56.880016 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:56.879982    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:01:56.976369 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:56.976333    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd" event={"ID":"ced35944-e749-49e0-97ac-806b824c601c","Type":"ContainerStarted","Data":"232c13f86c349523d3d1cc3640458822ef4d34ac3ae3a1ed54dba41656301e03"}
Oct 30 05:01:56.976818 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:56.976389    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd" event={"ID":"ced35944-e749-49e0-97ac-806b824c601c","Type":"ContainerStarted","Data":"433ff2fe513d5690e04c04e7c293e03bc8710e79900c859bf921d51fdc5f8427"}
Oct 30 05:01:57.018595 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:57.018540    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd" podStartSLOduration=5.018526096 podStartE2EDuration="5.018526096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:01:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:01:57.015058349 +0000 UTC m=+1046.238878283" watchObservedRunningTime="2026-10-30 05:01:57.018526096 +0000 UTC m=+1046.242345998"
Oct 30 05:01:57.721801 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:57.721767    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd"
Oct 30 05:01:57.723456 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:57.723429    5898 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 05:01:57.723587 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:57.723490    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd" podUID="ced35944-e749-49e0-97ac-806b824c601c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 05:01:58.722512 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:58.722475    5898 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 05:01:58.722928 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:58.722534    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd" podUID="ced35944-e749-49e0-97ac-806b824c601c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 05:01:58.851608 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:58.851557    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-566b2bc6-hbgtv"]
Oct 30 05:01:58.855880 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:58.855864    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-566b2bc6-hbgtv"
Oct 30 05:01:58.882197 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:58.882174    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:01:58.882537 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:58.882512    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\""
Oct 30 05:01:58.917805 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:58.917775    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-566b2bc6-hbgtv"]
Oct 30 05:01:58.922736 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:58.922702    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbqdz\" (UniqueName: \"kubernetes.io/projected/25bc3551-4714-473b-815a-2844fbdcdb59-kube-api-access-jbqdz\") pod \"maas-api-revoke-sub-566b2bc6-hbgtv\" (UID: \"25bc3551-4714-473b-815a-2844fbdcdb59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-566b2bc6-hbgtv"
Oct 30 05:01:58.924951 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:58.924922    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25bc3551-4714-473b-815a-2844fbdcdb59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-566b2bc6-hbgtv\" (UID: \"25bc3551-4714-473b-815a-2844fbdcdb59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-566b2bc6-hbgtv"
Oct 30 05:01:58.925195 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:58.925177    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25bc3551-4714-473b-815a-2844fbdcdb59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-566b2bc6-hbgtv\" (UID: \"25bc3551-4714-473b-815a-2844fbdcdb59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-566b2bc6-hbgtv"
Oct 30 05:01:59.026315 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:59.026232    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbqdz\" (UniqueName: \"kubernetes.io/projected/25bc3551-4714-473b-815a-2844fbdcdb59-kube-api-access-jbqdz\") pod \"maas-api-revoke-sub-566b2bc6-hbgtv\" (UID: \"25bc3551-4714-473b-815a-2844fbdcdb59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-566b2bc6-hbgtv"
Oct 30 05:01:59.026315 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:59.026274    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25bc3551-4714-473b-815a-2844fbdcdb59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-566b2bc6-hbgtv\" (UID: \"25bc3551-4714-473b-815a-2844fbdcdb59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-566b2bc6-hbgtv"
Oct 30 05:01:59.026528 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:59.026508    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25bc3551-4714-473b-815a-2844fbdcdb59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-566b2bc6-hbgtv\" (UID: \"25bc3551-4714-473b-815a-2844fbdcdb59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-566b2bc6-hbgtv"
Oct 30 05:01:59.027340 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:59.027321    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25bc3551-4714-473b-815a-2844fbdcdb59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-566b2bc6-hbgtv\" (UID: \"25bc3551-4714-473b-815a-2844fbdcdb59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-566b2bc6-hbgtv"
Oct 30 05:01:59.028610 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:59.028591    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25bc3551-4714-473b-815a-2844fbdcdb59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-566b2bc6-hbgtv\" (UID: \"25bc3551-4714-473b-815a-2844fbdcdb59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-566b2bc6-hbgtv"
Oct 30 05:01:59.045254 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:59.045234    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbqdz\" (UniqueName: \"kubernetes.io/projected/25bc3551-4714-473b-815a-2844fbdcdb59-kube-api-access-jbqdz\") pod \"maas-api-revoke-sub-566b2bc6-hbgtv\" (UID: \"25bc3551-4714-473b-815a-2844fbdcdb59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-566b2bc6-hbgtv"
Oct 30 05:01:59.169696 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:59.169667    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-566b2bc6-hbgtv"
Oct 30 05:01:59.404577 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:59.404536    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:01:59.514673 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:59.514642    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-566b2bc6-hbgtv"]
Oct 30 05:01:59.517512 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:01:59.517124    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25bc3551_4714_473b_815a_2844fbdcdb59.slice/crio-b1e0320371bc246dfe94c6cb145adfdc091e245f13e18f076e5149a1666f3180 WatchSource:0}: Error finding container b1e0320371bc246dfe94c6cb145adfdc091e245f13e18f076e5149a1666f3180: Status 404 returned error can't find the container with id b1e0320371bc246dfe94c6cb145adfdc091e245f13e18f076e5149a1666f3180
Oct 30 05:01:59.722509 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:59.722482    5898 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 05:01:59.722665 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:01:59.722529    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd" podUID="ced35944-e749-49e0-97ac-806b824c601c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 05:02:00.023502 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:00.023402    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-566b2bc6-hbgtv" event={"ID":"25bc3551-4714-473b-815a-2844fbdcdb59","Type":"ContainerStarted","Data":"484e62f784148bd6b6ce30ac4f8d4d3536e0e4555542c2fb2b368e9a53f36f1a"}
Oct 30 05:02:00.023502 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:00.023467    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-566b2bc6-hbgtv" event={"ID":"25bc3551-4714-473b-815a-2844fbdcdb59","Type":"ContainerStarted","Data":"b1e0320371bc246dfe94c6cb145adfdc091e245f13e18f076e5149a1666f3180"}
Oct 30 05:02:00.064207 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:00.064150    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-566b2bc6-hbgtv" podStartSLOduration=2.06413205 podStartE2EDuration="2.06413205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:01:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:02:00.059291592 +0000 UTC m=+1049.283111493" watchObservedRunningTime="2026-10-30 05:02:00.06413205 +0000 UTC m=+1049.287951952"
Oct 30 05:02:00.722158 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:00.722125    5898 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 05:02:00.722345 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:00.722176    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd" podUID="ced35944-e749-49e0-97ac-806b824c601c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 05:02:01.033462 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:01.033379    5898 generic.go:356] "Generic (PLEG): container finished" podID="25bc3551-4714-473b-815a-2844fbdcdb59" containerID="484e62f784148bd6b6ce30ac4f8d4d3536e0e4555542c2fb2b368e9a53f36f1a" exitCode=0
Oct 30 05:02:01.033895 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:01.033457    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-566b2bc6-hbgtv" event={"ID":"25bc3551-4714-473b-815a-2844fbdcdb59","Type":"ContainerDied","Data":"484e62f784148bd6b6ce30ac4f8d4d3536e0e4555542c2fb2b368e9a53f36f1a"}
Oct 30 05:02:01.090925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:01.089397    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:02:01.722312 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:01.722275    5898 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 05:02:01.722528 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:01.722341    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd" podUID="ced35944-e749-49e0-97ac-806b824c601c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 05:02:02.253199 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:02.253171    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-566b2bc6-hbgtv"
Oct 30 05:02:02.372839 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:02.372811    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25bc3551-4714-473b-815a-2844fbdcdb59-kube-api-access-jbqdz\" (UniqueName: \"kubernetes.io/projected/25bc3551-4714-473b-815a-2844fbdcdb59-kube-api-access-jbqdz\") pod \"25bc3551-4714-473b-815a-2844fbdcdb59\" (UID: \"25bc3551-4714-473b-815a-2844fbdcdb59\") "
Oct 30 05:02:02.373052 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:02.372985    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25bc3551-4714-473b-815a-2844fbdcdb59-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25bc3551-4714-473b-815a-2844fbdcdb59-maas-api-xks-ca\") pod \"25bc3551-4714-473b-815a-2844fbdcdb59\" (UID: \"25bc3551-4714-473b-815a-2844fbdcdb59\") "
Oct 30 05:02:02.373112 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:02.373074    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25bc3551-4714-473b-815a-2844fbdcdb59-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25bc3551-4714-473b-815a-2844fbdcdb59-maas-api-openshift-ca\") pod \"25bc3551-4714-473b-815a-2844fbdcdb59\" (UID: \"25bc3551-4714-473b-815a-2844fbdcdb59\") "
Oct 30 05:02:02.373471 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:02.373445    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25bc3551-4714-473b-815a-2844fbdcdb59-maas-api-openshift-ca" pod "25bc3551-4714-473b-815a-2844fbdcdb59" (UID: "25bc3551-4714-473b-815a-2844fbdcdb59"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:02:02.373622 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:02.373607    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25bc3551-4714-473b-815a-2844fbdcdb59-maas-api-openshift-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:02:02.385922 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:02.375341    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25bc3551-4714-473b-815a-2844fbdcdb59-kube-api-access-jbqdz" pod "25bc3551-4714-473b-815a-2844fbdcdb59" (UID: "25bc3551-4714-473b-815a-2844fbdcdb59"). InnerVolumeSpecName "kube-api-access-jbqdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:02:02.389920 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:02.389290    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25bc3551-4714-473b-815a-2844fbdcdb59-maas-api-xks-ca" pod "25bc3551-4714-473b-815a-2844fbdcdb59" (UID: "25bc3551-4714-473b-815a-2844fbdcdb59"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:02:02.479926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:02.479086    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbqdz\" (UniqueName: \"kubernetes.io/projected/25bc3551-4714-473b-815a-2844fbdcdb59-kube-api-access-jbqdz\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:02:02.479926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:02.479124    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25bc3551-4714-473b-815a-2844fbdcdb59-maas-api-xks-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:02:02.722198 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:02.722121    5898 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 05:02:02.722198 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:02.722187    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd" podUID="ced35944-e749-49e0-97ac-806b824c601c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 05:02:03.051624 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:03.051538    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-566b2bc6-hbgtv"
Oct 30 05:02:03.051624 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:03.051554    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-566b2bc6-hbgtv" event={"ID":"25bc3551-4714-473b-815a-2844fbdcdb59","Type":"ContainerDied","Data":"b1e0320371bc246dfe94c6cb145adfdc091e245f13e18f076e5149a1666f3180"}
Oct 30 05:02:03.051624 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:03.051611    5898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1e0320371bc246dfe94c6cb145adfdc091e245f13e18f076e5149a1666f3180"
Oct 30 05:02:03.079819 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:03.079775    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:02:03.722051 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:03.722017    5898 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 05:02:03.722554 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:03.722066    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd" podUID="ced35944-e749-49e0-97ac-806b824c601c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 05:02:04.726721 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:04.726690    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd"
Oct 30 05:02:04.727152 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:04.727114    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd"
Oct 30 05:02:04.727846 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:04.727827    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd"
Oct 30 05:02:04.947879 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:04.947839    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:02:06.764932 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:06.764881    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:02:08.459287 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:08.459254    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:02:10.259409 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:10.259377    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:02:11.960357 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:11.960321    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:02:13.788689 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:13.788654    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:02:15.651283 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:15.651249    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:02:17.541500 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:17.541457    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:02:17.766095 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:17.766059    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46a9f75e-wbj7s"]
Oct 30 05:02:17.766634 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:17.766620    5898 cpu_manager.go:414] "containerMap: removing container" podUID="25bc3551-4714-473b-815a-2844fbdcdb59" containerName="revoke-keys"
Oct 30 05:02:17.766682 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:17.766635    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25bc3551-4714-473b-815a-2844fbdcdb59" containerName="revoke-keys"
Oct 30 05:02:17.766769 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:17.766758    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="25bc3551-4714-473b-815a-2844fbdcdb59" containerName="revoke-keys"
Oct 30 05:02:17.770346 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:17.770327    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46a9f75e-wbj7s"
Oct 30 05:02:17.785397 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:17.785370    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46a9f75e-wbj7s"]
Oct 30 05:02:17.792401 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:17.792327    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:02:17.792401 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:17.792392    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\""
Oct 30 05:02:17.801495 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:17.801461    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/161f48e4-7e83-44ff-a12b-4f77d53dc5a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46a9f75e-wbj7s\" (UID: \"161f48e4-7e83-44ff-a12b-4f77d53dc5a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46a9f75e-wbj7s"
Oct 30 05:02:17.801623 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:17.801564    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kps8z\" (UniqueName: \"kubernetes.io/projected/161f48e4-7e83-44ff-a12b-4f77d53dc5a8-kube-api-access-kps8z\") pod \"maas-api-revoke-sub-46a9f75e-wbj7s\" (UID: \"161f48e4-7e83-44ff-a12b-4f77d53dc5a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46a9f75e-wbj7s"
Oct 30 05:02:17.801827 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:17.801799    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/161f48e4-7e83-44ff-a12b-4f77d53dc5a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46a9f75e-wbj7s\" (UID: \"161f48e4-7e83-44ff-a12b-4f77d53dc5a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46a9f75e-wbj7s"
Oct 30 05:02:17.904375 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:17.904341    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/161f48e4-7e83-44ff-a12b-4f77d53dc5a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46a9f75e-wbj7s\" (UID: \"161f48e4-7e83-44ff-a12b-4f77d53dc5a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46a9f75e-wbj7s"
Oct 30 05:02:17.904591 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:17.904495    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/161f48e4-7e83-44ff-a12b-4f77d53dc5a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46a9f75e-wbj7s\" (UID: \"161f48e4-7e83-44ff-a12b-4f77d53dc5a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46a9f75e-wbj7s"
Oct 30 05:02:17.904591 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:17.904558    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kps8z\" (UniqueName: \"kubernetes.io/projected/161f48e4-7e83-44ff-a12b-4f77d53dc5a8-kube-api-access-kps8z\") pod \"maas-api-revoke-sub-46a9f75e-wbj7s\" (UID: \"161f48e4-7e83-44ff-a12b-4f77d53dc5a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46a9f75e-wbj7s"
Oct 30 05:02:17.905302 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:17.905267    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/161f48e4-7e83-44ff-a12b-4f77d53dc5a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46a9f75e-wbj7s\" (UID: \"161f48e4-7e83-44ff-a12b-4f77d53dc5a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46a9f75e-wbj7s"
Oct 30 05:02:17.907469 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:17.907447    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/161f48e4-7e83-44ff-a12b-4f77d53dc5a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46a9f75e-wbj7s\" (UID: \"161f48e4-7e83-44ff-a12b-4f77d53dc5a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46a9f75e-wbj7s"
Oct 30 05:02:17.923362 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:17.923337    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kps8z\" (UniqueName: \"kubernetes.io/projected/161f48e4-7e83-44ff-a12b-4f77d53dc5a8-kube-api-access-kps8z\") pod \"maas-api-revoke-sub-46a9f75e-wbj7s\" (UID: \"161f48e4-7e83-44ff-a12b-4f77d53dc5a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46a9f75e-wbj7s"
Oct 30 05:02:18.084861 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:18.084826    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46a9f75e-wbj7s"
Oct 30 05:02:18.384432 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:18.384398    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46a9f75e-wbj7s"]
Oct 30 05:02:18.384635 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:02:18.384613    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod161f48e4_7e83_44ff_a12b_4f77d53dc5a8.slice/crio-fcbb7c37bc9ab591fa930b736d0cb99e30b77b11b65eb480f318f079c0df01c9 WatchSource:0}: Error finding container fcbb7c37bc9ab591fa930b736d0cb99e30b77b11b65eb480f318f079c0df01c9: Status 404 returned error can't find the container with id fcbb7c37bc9ab591fa930b736d0cb99e30b77b11b65eb480f318f079c0df01c9
Oct 30 05:02:19.169983 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:19.169939    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46a9f75e-wbj7s" event={"ID":"161f48e4-7e83-44ff-a12b-4f77d53dc5a8","Type":"ContainerStarted","Data":"7fbcc6327c15f43e1f3e1652793951571925438f19b9c1ce865e35c9349d6917"}
Oct 30 05:02:19.170421 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:19.169999    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46a9f75e-wbj7s" event={"ID":"161f48e4-7e83-44ff-a12b-4f77d53dc5a8","Type":"ContainerStarted","Data":"fcbb7c37bc9ab591fa930b736d0cb99e30b77b11b65eb480f318f079c0df01c9"}
Oct 30 05:02:19.209091 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:19.209043    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46a9f75e-wbj7s" podStartSLOduration=2.209029648 podStartE2EDuration="2.209029648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:02:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:02:19.205764324 +0000 UTC m=+1068.429584222" watchObservedRunningTime="2026-10-30 05:02:19.209029648 +0000 UTC m=+1068.432849552"
Oct 30 05:02:19.383360 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:19.383314    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:02:20.259925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:20.259351    5898 generic.go:356] "Generic (PLEG): container finished" podID="161f48e4-7e83-44ff-a12b-4f77d53dc5a8" containerID="7fbcc6327c15f43e1f3e1652793951571925438f19b9c1ce865e35c9349d6917" exitCode=0
Oct 30 05:02:20.259925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:20.259579    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46a9f75e-wbj7s" event={"ID":"161f48e4-7e83-44ff-a12b-4f77d53dc5a8","Type":"ContainerDied","Data":"7fbcc6327c15f43e1f3e1652793951571925438f19b9c1ce865e35c9349d6917"}
Oct 30 05:02:21.248093 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:21.248048    5898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6" podUID="47ff272a-ac5c-4f59-bf7b-5c0692d09386" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:21.550977 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:21.544032    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g" podUID="70294a85-1050-4632-b3bf-c3e44a4c0b41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:21.550977 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:21.544429    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5" podUID="ebf8f4a2-8ac6-48ec-bdba-3f6a99653264" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 05:02:21.550977 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:21.546099    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6" podUID="47ff272a-ac5c-4f59-bf7b-5c0692d09386" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:21.571146 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:21.571099    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9" podUID="1647f909-5580-4219-a2e1-30b4cd994379" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:21.627924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:21.609652    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb" podUID="6f16bd6a-a7f0-49bb-844f-57b777bde2ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 05:02:21.676135 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:21.676094    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49" podUID="05eef4eb-f379-4dc7-a947-123bd85a4968" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 05:02:21.899083 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:21.899029    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6" podUID="b09804e3-8135-4f6d-ae34-c45d6da9728e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:21.899769 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:21.899360    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh" podUID="38428a53-04c7-400a-9c78-52407e982478" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 30 05:02:21.945926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:21.910089    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58" podUID="c27f3b21-1447-4af9-8d99-c04fa9b23cda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:22.029421 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:21.948817    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849" podUID="a9707f50-d5e1-409f-b8f3-c58629677aaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:22.029421 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:21.962981    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247" podUID="1d865d3e-a6e9-407e-844e-cc75fc7f1354" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:22.312797 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:22.312693    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx" podUID="d523008f-c740-4a41-89cb-e82e1ffa0094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 05:02:22.578009 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:22.570017    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6" podUID="47ff272a-ac5c-4f59-bf7b-5c0692d09386" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:22.625925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:22.610068    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb" podUID="6f16bd6a-a7f0-49bb-844f-57b777bde2ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:22.625925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:22.584182    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5" podUID="ebf8f4a2-8ac6-48ec-bdba-3f6a99653264" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 05:02:22.629922 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:22.628947    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6"
Oct 30 05:02:22.629922 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:22.629055    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb"
Oct 30 05:02:22.644925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:22.605667    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9" podUID="1647f909-5580-4219-a2e1-30b4cd994379" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:22.658054 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:22.658022    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5"
Oct 30 05:02:22.658346 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:22.658329    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9"
Oct 30 05:02:22.750410 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:22.735000    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49" podUID="05eef4eb-f379-4dc7-a947-123bd85a4968" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:22.750410 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:22.735410    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49"
Oct 30 05:02:22.919523 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:22.908982    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6" podUID="b09804e3-8135-4f6d-ae34-c45d6da9728e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 05:02:22.988927 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:22.975502    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849" podUID="a9707f50-d5e1-409f-b8f3-c58629677aaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:22.988927 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:22.977368    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247" podUID="1d865d3e-a6e9-407e-844e-cc75fc7f1354" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:22.988927 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:22.978712    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh" podUID="38428a53-04c7-400a-9c78-52407e982478" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:23.028950 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:23.024416    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58" podUID="c27f3b21-1447-4af9-8d99-c04fa9b23cda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:23.236975 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:23.210174    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6"
Oct 30 05:02:23.238423 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:23.238394    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849"
Oct 30 05:02:23.238631 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:23.238616    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247"
Oct 30 05:02:23.238768 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:23.238756    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh"
Oct 30 05:02:23.238921 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:23.238890    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58"
Oct 30 05:02:23.480502 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:23.448350    5898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx" podUID="d523008f-c740-4a41-89cb-e82e1ffa0094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:23.480502 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:23.460030    5898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247" podUID="1d865d3e-a6e9-407e-844e-cc75fc7f1354" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": context deadline exceeded"
Oct 30 05:02:23.480502 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:23.460118    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx" podUID="d523008f-c740-4a41-89cb-e82e1ffa0094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 05:02:23.480502 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:23.460361    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx"
Oct 30 05:02:23.480502 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:23.477232    5898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb" podUID="6f16bd6a-a7f0-49bb-844f-57b777bde2ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:23.515849 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:23.515679    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46a9f75e-wbj7s"
Oct 30 05:02:23.634192 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:23.634063    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g" podUID="70294a85-1050-4632-b3bf-c3e44a4c0b41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 05:02:23.644112 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:23.644069    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb" podUID="6f16bd6a-a7f0-49bb-844f-57b777bde2ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 05:02:23.645103 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:23.645074    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5" podUID="ebf8f4a2-8ac6-48ec-bdba-3f6a99653264" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 05:02:23.679301 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:23.679253    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9" podUID="1647f909-5580-4219-a2e1-30b4cd994379" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:23.715140 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:23.715098    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6" podUID="47ff272a-ac5c-4f59-bf7b-5c0692d09386" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:23.763922 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:23.763226    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49" podUID="05eef4eb-f379-4dc7-a947-123bd85a4968" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:23.959269 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:23.959222    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6" podUID="b09804e3-8135-4f6d-ae34-c45d6da9728e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:23.977428 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:23.977393    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/161f48e4-7e83-44ff-a12b-4f77d53dc5a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/161f48e4-7e83-44ff-a12b-4f77d53dc5a8-maas-api-openshift-ca\") pod \"161f48e4-7e83-44ff-a12b-4f77d53dc5a8\" (UID: \"161f48e4-7e83-44ff-a12b-4f77d53dc5a8\") "
Oct 30 05:02:24.031003 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:24.028979    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58" podUID="c27f3b21-1447-4af9-8d99-c04fa9b23cda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:24.034061 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:23.978502    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849" podUID="a9707f50-d5e1-409f-b8f3-c58629677aaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 05:02:24.035621 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:23.978454    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247" podUID="1d865d3e-a6e9-407e-844e-cc75fc7f1354" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:24.035621 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:24.035131    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/161f48e4-7e83-44ff-a12b-4f77d53dc5a8-maas-api-openshift-ca" pod "161f48e4-7e83-44ff-a12b-4f77d53dc5a8" (UID: "161f48e4-7e83-44ff-a12b-4f77d53dc5a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:02:24.036477 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:24.036440    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/161f48e4-7e83-44ff-a12b-4f77d53dc5a8-kube-api-access-kps8z\" (UniqueName: \"kubernetes.io/projected/161f48e4-7e83-44ff-a12b-4f77d53dc5a8-kube-api-access-kps8z\") pod \"161f48e4-7e83-44ff-a12b-4f77d53dc5a8\" (UID: \"161f48e4-7e83-44ff-a12b-4f77d53dc5a8\") "
Oct 30 05:02:24.037327 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:24.037296    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/161f48e4-7e83-44ff-a12b-4f77d53dc5a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/161f48e4-7e83-44ff-a12b-4f77d53dc5a8-maas-api-xks-ca\") pod \"161f48e4-7e83-44ff-a12b-4f77d53dc5a8\" (UID: \"161f48e4-7e83-44ff-a12b-4f77d53dc5a8\") "
Oct 30 05:02:24.135437 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:24.135396    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh" podUID="38428a53-04c7-400a-9c78-52407e982478" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:24.194926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:24.146401    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/161f48e4-7e83-44ff-a12b-4f77d53dc5a8-maas-api-openshift-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:02:24.343926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:24.343723    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46a9f75e-wbj7s" event={"ID":"161f48e4-7e83-44ff-a12b-4f77d53dc5a8","Type":"ContainerDied","Data":"fcbb7c37bc9ab591fa930b736d0cb99e30b77b11b65eb480f318f079c0df01c9"}
Oct 30 05:02:24.343926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:24.343810    5898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fcbb7c37bc9ab591fa930b736d0cb99e30b77b11b65eb480f318f079c0df01c9"
Oct 30 05:02:24.344190 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:24.344011    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46a9f75e-wbj7s"
Oct 30 05:02:24.569945 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:24.567696    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx" podUID="d523008f-c740-4a41-89cb-e82e1ffa0094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 05:02:24.657712 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:24.657613    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5" podUID="ebf8f4a2-8ac6-48ec-bdba-3f6a99653264" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:24.679927 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:24.663321    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g" podUID="70294a85-1050-4632-b3bf-c3e44a4c0b41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 05:02:24.681549 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:24.681511    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb" podUID="6f16bd6a-a7f0-49bb-844f-57b777bde2ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 05:02:24.685924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:24.682167    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g"
Oct 30 05:02:24.713927 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:24.688731    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9" podUID="1647f909-5580-4219-a2e1-30b4cd994379" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:25.152924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:24.742724    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6" podUID="47ff272a-ac5c-4f59-bf7b-5c0692d09386" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:25.152924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:24.782213    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49" podUID="05eef4eb-f379-4dc7-a947-123bd85a4968" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:25.152924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:25.021710    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6" podUID="b09804e3-8135-4f6d-ae34-c45d6da9728e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 05:02:25.152924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:25.029143    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58" podUID="c27f3b21-1447-4af9-8d99-c04fa9b23cda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:25.152924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:25.031369    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849" podUID="a9707f50-d5e1-409f-b8f3-c58629677aaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:25.152924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:25.074858    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/161f48e4-7e83-44ff-a12b-4f77d53dc5a8-kube-api-access-kps8z" pod "161f48e4-7e83-44ff-a12b-4f77d53dc5a8" (UID: "161f48e4-7e83-44ff-a12b-4f77d53dc5a8"). InnerVolumeSpecName "kube-api-access-kps8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:02:25.152924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:25.120359    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/161f48e4-7e83-44ff-a12b-4f77d53dc5a8-maas-api-xks-ca" pod "161f48e4-7e83-44ff-a12b-4f77d53dc5a8" (UID: "161f48e4-7e83-44ff-a12b-4f77d53dc5a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:02:25.152924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:25.102570    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247" podUID="1d865d3e-a6e9-407e-844e-cc75fc7f1354" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 05:02:25.152924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:25.146077    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh" podUID="38428a53-04c7-400a-9c78-52407e982478" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:25.203715 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:25.203678    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kps8z\" (UniqueName: \"kubernetes.io/projected/161f48e4-7e83-44ff-a12b-4f77d53dc5a8-kube-api-access-kps8z\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:02:25.263166 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:25.263133    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/161f48e4-7e83-44ff-a12b-4f77d53dc5a8-maas-api-xks-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:02:25.589221 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:25.589181    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx" podUID="d523008f-c740-4a41-89cb-e82e1ffa0094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:25.677003 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:25.676952    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5" podUID="ebf8f4a2-8ac6-48ec-bdba-3f6a99653264" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:25.690956 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:25.681657    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g" podUID="70294a85-1050-4632-b3bf-c3e44a4c0b41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:25.706006 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:25.705962    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb" podUID="6f16bd6a-a7f0-49bb-844f-57b777bde2ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:26.162666 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:26.141111    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6" podUID="b09804e3-8135-4f6d-ae34-c45d6da9728e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:26.162666 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:26.141510    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9" podUID="1647f909-5580-4219-a2e1-30b4cd994379" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:26.162666 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:26.141784    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6" podUID="47ff272a-ac5c-4f59-bf7b-5c0692d09386" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Oct 30 05:02:26.162666 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:26.142043    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49" podUID="05eef4eb-f379-4dc7-a947-123bd85a4968" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 05:02:26.162666 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:26.142649    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849" podUID="a9707f50-d5e1-409f-b8f3-c58629677aaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:26.162666 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:26.161474    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh" podUID="38428a53-04c7-400a-9c78-52407e982478" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:26.162666 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:26.161842    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247" podUID="1d865d3e-a6e9-407e-844e-cc75fc7f1354" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:26.162666 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:26.162269    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58" podUID="c27f3b21-1447-4af9-8d99-c04fa9b23cda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:26.201286 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:26.201257    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49"
Oct 30 05:02:26.291430 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:26.291398    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9"
Oct 30 05:02:26.311986 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:26.311956    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:02:26.330987 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:26.330962    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849"
Oct 30 05:02:26.589537 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:26.589497    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx" podUID="d523008f-c740-4a41-89cb-e82e1ffa0094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:26.591578 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:26.590392    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6"
Oct 30 05:02:26.652860 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:26.652832    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx"
Oct 30 05:02:26.722919 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:26.721081    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g" podUID="70294a85-1050-4632-b3bf-c3e44a4c0b41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:26.723404 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:26.723287    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb" podUID="6f16bd6a-a7f0-49bb-844f-57b777bde2ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 05:02:26.748601 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:26.729915    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5" podUID="ebf8f4a2-8ac6-48ec-bdba-3f6a99653264" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 05:02:26.748601 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:26.745664    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247"
Oct 30 05:02:26.771807 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:26.771570    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6"
Oct 30 05:02:26.772827 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:26.772782    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb"
Oct 30 05:02:26.788381 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:26.788360    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5"
Oct 30 05:02:26.865867 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:26.865181    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh"
Oct 30 05:02:26.939682 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:02:26.939646    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:02:26.953944 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:26.943813    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g"
Oct 30 05:02:26.953944 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:26.943895    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58"
Oct 30 05:02:29.060795 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:29.060762    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:02:29.242087 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:29.242045    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f2f688d-sw2jc"]
Oct 30 05:02:29.242634 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:29.242620    5898 cpu_manager.go:414] "containerMap: removing container" podUID="161f48e4-7e83-44ff-a12b-4f77d53dc5a8" containerName="revoke-keys"
Oct 30 05:02:29.242680 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:29.242636    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="161f48e4-7e83-44ff-a12b-4f77d53dc5a8" containerName="revoke-keys"
Oct 30 05:02:29.242738 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:29.242730    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="161f48e4-7e83-44ff-a12b-4f77d53dc5a8" containerName="revoke-keys"
Oct 30 05:02:29.246471 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:29.246450    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f2f688d-sw2jc"
Oct 30 05:02:29.263915 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:29.263876    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f2f688d-sw2jc"]
Oct 30 05:02:29.270828 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:29.270804    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:02:29.270828 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:29.270817    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\""
Oct 30 05:02:29.348966 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:29.348935    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6366537-f230-4971-b69b-ac4ffd5eb4c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f2f688d-sw2jc\" (UID: \"f6366537-f230-4971-b69b-ac4ffd5eb4c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f2f688d-sw2jc"
Oct 30 05:02:29.349111 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:29.348990    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6366537-f230-4971-b69b-ac4ffd5eb4c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f2f688d-sw2jc\" (UID: \"f6366537-f230-4971-b69b-ac4ffd5eb4c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f2f688d-sw2jc"
Oct 30 05:02:29.349159 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:29.349121    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl6jm\" (UniqueName: \"kubernetes.io/projected/f6366537-f230-4971-b69b-ac4ffd5eb4c5-kube-api-access-hl6jm\") pod \"maas-api-revoke-sub-0f2f688d-sw2jc\" (UID: \"f6366537-f230-4971-b69b-ac4ffd5eb4c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f2f688d-sw2jc"
Oct 30 05:02:29.450326 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:29.450294    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6366537-f230-4971-b69b-ac4ffd5eb4c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f2f688d-sw2jc\" (UID: \"f6366537-f230-4971-b69b-ac4ffd5eb4c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f2f688d-sw2jc"
Oct 30 05:02:29.450519 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:29.450344    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6366537-f230-4971-b69b-ac4ffd5eb4c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f2f688d-sw2jc\" (UID: \"f6366537-f230-4971-b69b-ac4ffd5eb4c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f2f688d-sw2jc"
Oct 30 05:02:29.450519 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:29.450446    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl6jm\" (UniqueName: \"kubernetes.io/projected/f6366537-f230-4971-b69b-ac4ffd5eb4c5-kube-api-access-hl6jm\") pod \"maas-api-revoke-sub-0f2f688d-sw2jc\" (UID: \"f6366537-f230-4971-b69b-ac4ffd5eb4c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f2f688d-sw2jc"
Oct 30 05:02:29.450985 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:29.450962    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6366537-f230-4971-b69b-ac4ffd5eb4c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f2f688d-sw2jc\" (UID: \"f6366537-f230-4971-b69b-ac4ffd5eb4c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f2f688d-sw2jc"
Oct 30 05:02:29.452727 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:29.452706    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6366537-f230-4971-b69b-ac4ffd5eb4c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f2f688d-sw2jc\" (UID: \"f6366537-f230-4971-b69b-ac4ffd5eb4c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f2f688d-sw2jc"
Oct 30 05:02:29.470109 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:29.470083    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl6jm\" (UniqueName: \"kubernetes.io/projected/f6366537-f230-4971-b69b-ac4ffd5eb4c5-kube-api-access-hl6jm\") pod \"maas-api-revoke-sub-0f2f688d-sw2jc\" (UID: \"f6366537-f230-4971-b69b-ac4ffd5eb4c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f2f688d-sw2jc"
Oct 30 05:02:29.560485 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:29.560452    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f2f688d-sw2jc"
Oct 30 05:02:29.858969 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:29.858936    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f2f688d-sw2jc"]
Oct 30 05:02:29.859140 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:02:29.859054    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6366537_f230_4971_b69b_ac4ffd5eb4c5.slice/crio-1f2a89ce18474c2cd64d88f66f3057de3f586f79565b5cda46d0ab528bfb17e7 WatchSource:0}: Error finding container 1f2a89ce18474c2cd64d88f66f3057de3f586f79565b5cda46d0ab528bfb17e7: Status 404 returned error can't find the container with id 1f2a89ce18474c2cd64d88f66f3057de3f586f79565b5cda46d0ab528bfb17e7
Oct 30 05:02:30.554045 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:30.553948    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f2f688d-sw2jc" event={"ID":"f6366537-f230-4971-b69b-ac4ffd5eb4c5","Type":"ContainerStarted","Data":"4a692d258bf70e73862e8a79b3ecdae74a516fb5f39128bee72338b91c4e1c60"}
Oct 30 05:02:30.554413 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:30.554052    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f2f688d-sw2jc" event={"ID":"f6366537-f230-4971-b69b-ac4ffd5eb4c5","Type":"ContainerStarted","Data":"1f2a89ce18474c2cd64d88f66f3057de3f586f79565b5cda46d0ab528bfb17e7"}
Oct 30 05:02:30.589389 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:30.589339    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f2f688d-sw2jc" podStartSLOduration=1.589328533 podStartE2EDuration="1.589328533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:02:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:02:30.589005362 +0000 UTC m=+1079.812825263" watchObservedRunningTime="2026-10-30 05:02:30.589328533 +0000 UTC m=+1079.813148434"
Oct 30 05:02:31.067888 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:31.067845    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:02:31.562489 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:31.562457    5898 generic.go:356] "Generic (PLEG): container finished" podID="f6366537-f230-4971-b69b-ac4ffd5eb4c5" containerID="4a692d258bf70e73862e8a79b3ecdae74a516fb5f39128bee72338b91c4e1c60" exitCode=0
Oct 30 05:02:31.562882 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:31.562537    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f2f688d-sw2jc" event={"ID":"f6366537-f230-4971-b69b-ac4ffd5eb4c5","Type":"ContainerDied","Data":"4a692d258bf70e73862e8a79b3ecdae74a516fb5f39128bee72338b91c4e1c60"}
Oct 30 05:02:32.723083 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:32.723057    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f2f688d-sw2jc"
Oct 30 05:02:32.766710 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:32.766665    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:02:32.916273 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:32.916243    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6366537-f230-4971-b69b-ac4ffd5eb4c5-kube-api-access-hl6jm\" (UniqueName: \"kubernetes.io/projected/f6366537-f230-4971-b69b-ac4ffd5eb4c5-kube-api-access-hl6jm\") pod \"f6366537-f230-4971-b69b-ac4ffd5eb4c5\" (UID: \"f6366537-f230-4971-b69b-ac4ffd5eb4c5\") "
Oct 30 05:02:32.916465 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:32.916282    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6366537-f230-4971-b69b-ac4ffd5eb4c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6366537-f230-4971-b69b-ac4ffd5eb4c5-maas-api-xks-ca\") pod \"f6366537-f230-4971-b69b-ac4ffd5eb4c5\" (UID: \"f6366537-f230-4971-b69b-ac4ffd5eb4c5\") "
Oct 30 05:02:32.916547 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:32.916468    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6366537-f230-4971-b69b-ac4ffd5eb4c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6366537-f230-4971-b69b-ac4ffd5eb4c5-maas-api-openshift-ca\") pod \"f6366537-f230-4971-b69b-ac4ffd5eb4c5\" (UID: \"f6366537-f230-4971-b69b-ac4ffd5eb4c5\") "
Oct 30 05:02:32.916867 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:32.916845    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6366537-f230-4971-b69b-ac4ffd5eb4c5-maas-api-openshift-ca" pod "f6366537-f230-4971-b69b-ac4ffd5eb4c5" (UID: "f6366537-f230-4971-b69b-ac4ffd5eb4c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:02:32.917038 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:32.917021    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6366537-f230-4971-b69b-ac4ffd5eb4c5-maas-api-openshift-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:02:32.918373 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:32.918350    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6366537-f230-4971-b69b-ac4ffd5eb4c5-maas-api-xks-ca" pod "f6366537-f230-4971-b69b-ac4ffd5eb4c5" (UID: "f6366537-f230-4971-b69b-ac4ffd5eb4c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:02:32.918373 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:32.918360    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6366537-f230-4971-b69b-ac4ffd5eb4c5-kube-api-access-hl6jm" pod "f6366537-f230-4971-b69b-ac4ffd5eb4c5" (UID: "f6366537-f230-4971-b69b-ac4ffd5eb4c5"). InnerVolumeSpecName "kube-api-access-hl6jm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:02:33.018674 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:33.018642    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hl6jm\" (UniqueName: \"kubernetes.io/projected/f6366537-f230-4971-b69b-ac4ffd5eb4c5-kube-api-access-hl6jm\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:02:33.018674 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:33.018671    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6366537-f230-4971-b69b-ac4ffd5eb4c5-maas-api-xks-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:02:33.577282 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:33.577248    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f2f688d-sw2jc"
Oct 30 05:02:33.577477 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:33.577295    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f2f688d-sw2jc" event={"ID":"f6366537-f230-4971-b69b-ac4ffd5eb4c5","Type":"ContainerDied","Data":"1f2a89ce18474c2cd64d88f66f3057de3f586f79565b5cda46d0ab528bfb17e7"}
Oct 30 05:02:33.577477 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:33.577341    5898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f2a89ce18474c2cd64d88f66f3057de3f586f79565b5cda46d0ab528bfb17e7"
Oct 30 05:02:34.547728 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:34.547690    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:02:36.256954 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:36.256889    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:02:37.882213 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:37.882178    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed8db1e0-t48w9"]
Oct 30 05:02:37.882800 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:37.882784    5898 cpu_manager.go:414] "containerMap: removing container" podUID="f6366537-f230-4971-b69b-ac4ffd5eb4c5" containerName="revoke-keys"
Oct 30 05:02:37.882842 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:37.882801    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6366537-f230-4971-b69b-ac4ffd5eb4c5" containerName="revoke-keys"
Oct 30 05:02:37.882947 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:37.882937    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6366537-f230-4971-b69b-ac4ffd5eb4c5" containerName="revoke-keys"
Oct 30 05:02:37.886720 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:37.886700    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8db1e0-t48w9"
Oct 30 05:02:37.905472 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:37.905441    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed8db1e0-t48w9"]
Oct 30 05:02:37.911415 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:37.911389    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:02:37.911542 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:37.911532    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\""
Oct 30 05:02:37.986440 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:37.986409    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7804a436-264d-4812-8446-7ce83590a9b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed8db1e0-t48w9\" (UID: \"7804a436-264d-4812-8446-7ce83590a9b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8db1e0-t48w9"
Oct 30 05:02:37.986790 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:37.986769    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2gc7\" (UniqueName: \"kubernetes.io/projected/7804a436-264d-4812-8446-7ce83590a9b7-kube-api-access-f2gc7\") pod \"maas-api-revoke-sub-ed8db1e0-t48w9\" (UID: \"7804a436-264d-4812-8446-7ce83590a9b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8db1e0-t48w9"
Oct 30 05:02:37.986923 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:37.986887    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7804a436-264d-4812-8446-7ce83590a9b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed8db1e0-t48w9\" (UID: \"7804a436-264d-4812-8446-7ce83590a9b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8db1e0-t48w9"
Oct 30 05:02:38.015880 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:38.015851    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:02:38.091866 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:38.091809    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7804a436-264d-4812-8446-7ce83590a9b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed8db1e0-t48w9\" (UID: \"7804a436-264d-4812-8446-7ce83590a9b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8db1e0-t48w9"
Oct 30 05:02:38.092122 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:38.092095    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2gc7\" (UniqueName: \"kubernetes.io/projected/7804a436-264d-4812-8446-7ce83590a9b7-kube-api-access-f2gc7\") pod \"maas-api-revoke-sub-ed8db1e0-t48w9\" (UID: \"7804a436-264d-4812-8446-7ce83590a9b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8db1e0-t48w9"
Oct 30 05:02:38.092208 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:38.092191    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7804a436-264d-4812-8446-7ce83590a9b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed8db1e0-t48w9\" (UID: \"7804a436-264d-4812-8446-7ce83590a9b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8db1e0-t48w9"
Oct 30 05:02:38.093266 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:38.093240    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7804a436-264d-4812-8446-7ce83590a9b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed8db1e0-t48w9\" (UID: \"7804a436-264d-4812-8446-7ce83590a9b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8db1e0-t48w9"
Oct 30 05:02:38.111550 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:38.111514    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7804a436-264d-4812-8446-7ce83590a9b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed8db1e0-t48w9\" (UID: \"7804a436-264d-4812-8446-7ce83590a9b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8db1e0-t48w9"
Oct 30 05:02:38.116933 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:38.116873    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2gc7\" (UniqueName: \"kubernetes.io/projected/7804a436-264d-4812-8446-7ce83590a9b7-kube-api-access-f2gc7\") pod \"maas-api-revoke-sub-ed8db1e0-t48w9\" (UID: \"7804a436-264d-4812-8446-7ce83590a9b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8db1e0-t48w9"
Oct 30 05:02:38.200627 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:38.200542    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8db1e0-t48w9"
Oct 30 05:02:38.507129 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:02:38.507099    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7804a436_264d_4812_8446_7ce83590a9b7.slice/crio-760f7aa1332a96f12c5df550c1645663607bf8ac6482aa5c46f17071b0aa8bd8 WatchSource:0}: Error finding container 760f7aa1332a96f12c5df550c1645663607bf8ac6482aa5c46f17071b0aa8bd8: Status 404 returned error can't find the container with id 760f7aa1332a96f12c5df550c1645663607bf8ac6482aa5c46f17071b0aa8bd8
Oct 30 05:02:38.507707 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:38.507679    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed8db1e0-t48w9"]
Oct 30 05:02:38.611219 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:38.611191    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8db1e0-t48w9" event={"ID":"7804a436-264d-4812-8446-7ce83590a9b7","Type":"ContainerStarted","Data":"760f7aa1332a96f12c5df550c1645663607bf8ac6482aa5c46f17071b0aa8bd8"}
Oct 30 05:02:38.659883 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:38.659834    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:02:39.628184 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:39.628148    5898 generic.go:356] "Generic (PLEG): container finished" podID="7804a436-264d-4812-8446-7ce83590a9b7" containerID="33f9025b2af5244d87bec97ae4647509d69ff86c1f037723fa1ca8b12ed23826" exitCode=0
Oct 30 05:02:39.628591 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:39.628241    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8db1e0-t48w9" event={"ID":"7804a436-264d-4812-8446-7ce83590a9b7","Type":"ContainerDied","Data":"33f9025b2af5244d87bec97ae4647509d69ff86c1f037723fa1ca8b12ed23826"}
Oct 30 05:02:40.008061 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:40.007974    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:02:40.639747 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:40.639720    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6447008-d4gbk"]
Oct 30 05:02:40.644072 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:40.644049    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6447008-d4gbk"
Oct 30 05:02:40.665026 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:40.664989    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6447008-d4gbk"]
Oct 30 05:02:40.732379 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:40.732344    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a171566-49dd-491e-8f7b-a07f07c15453-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6447008-d4gbk\" (UID: \"5a171566-49dd-491e-8f7b-a07f07c15453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6447008-d4gbk"
Oct 30 05:02:40.732634 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:40.732537    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g75zg\" (UniqueName: \"kubernetes.io/projected/5a171566-49dd-491e-8f7b-a07f07c15453-kube-api-access-g75zg\") pod \"maas-api-revoke-sub-b6447008-d4gbk\" (UID: \"5a171566-49dd-491e-8f7b-a07f07c15453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6447008-d4gbk"
Oct 30 05:02:40.732712 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:40.732671    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a171566-49dd-491e-8f7b-a07f07c15453-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6447008-d4gbk\" (UID: \"5a171566-49dd-491e-8f7b-a07f07c15453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6447008-d4gbk"
Oct 30 05:02:40.833690 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:40.833663    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a171566-49dd-491e-8f7b-a07f07c15453-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6447008-d4gbk\" (UID: \"5a171566-49dd-491e-8f7b-a07f07c15453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6447008-d4gbk"
Oct 30 05:02:40.833824 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:40.833810    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g75zg\" (UniqueName: \"kubernetes.io/projected/5a171566-49dd-491e-8f7b-a07f07c15453-kube-api-access-g75zg\") pod \"maas-api-revoke-sub-b6447008-d4gbk\" (UID: \"5a171566-49dd-491e-8f7b-a07f07c15453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6447008-d4gbk"
Oct 30 05:02:40.833935 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:40.833888    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a171566-49dd-491e-8f7b-a07f07c15453-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6447008-d4gbk\" (UID: \"5a171566-49dd-491e-8f7b-a07f07c15453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6447008-d4gbk"
Oct 30 05:02:40.834480 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:40.834456    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a171566-49dd-491e-8f7b-a07f07c15453-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6447008-d4gbk\" (UID: \"5a171566-49dd-491e-8f7b-a07f07c15453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6447008-d4gbk"
Oct 30 05:02:40.836107 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:40.836084    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a171566-49dd-491e-8f7b-a07f07c15453-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6447008-d4gbk\" (UID: \"5a171566-49dd-491e-8f7b-a07f07c15453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6447008-d4gbk"
Oct 30 05:02:40.836259 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:40.836243    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8db1e0-t48w9"
Oct 30 05:02:40.852250 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:40.852224    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g75zg\" (UniqueName: \"kubernetes.io/projected/5a171566-49dd-491e-8f7b-a07f07c15453-kube-api-access-g75zg\") pod \"maas-api-revoke-sub-b6447008-d4gbk\" (UID: \"5a171566-49dd-491e-8f7b-a07f07c15453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6447008-d4gbk"
Oct 30 05:02:40.934610 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:40.934543    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7804a436-264d-4812-8446-7ce83590a9b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7804a436-264d-4812-8446-7ce83590a9b7-maas-api-xks-ca\") pod \"7804a436-264d-4812-8446-7ce83590a9b7\" (UID: \"7804a436-264d-4812-8446-7ce83590a9b7\") "
Oct 30 05:02:40.934767 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:40.934635    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7804a436-264d-4812-8446-7ce83590a9b7-kube-api-access-f2gc7\" (UniqueName: \"kubernetes.io/projected/7804a436-264d-4812-8446-7ce83590a9b7-kube-api-access-f2gc7\") pod \"7804a436-264d-4812-8446-7ce83590a9b7\" (UID: \"7804a436-264d-4812-8446-7ce83590a9b7\") "
Oct 30 05:02:40.934767 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:40.934670    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7804a436-264d-4812-8446-7ce83590a9b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7804a436-264d-4812-8446-7ce83590a9b7-maas-api-openshift-ca\") pod \"7804a436-264d-4812-8446-7ce83590a9b7\" (UID: \"7804a436-264d-4812-8446-7ce83590a9b7\") "
Oct 30 05:02:40.935341 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:40.935306    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7804a436-264d-4812-8446-7ce83590a9b7-maas-api-openshift-ca" pod "7804a436-264d-4812-8446-7ce83590a9b7" (UID: "7804a436-264d-4812-8446-7ce83590a9b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:02:40.936849 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:40.936823    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7804a436-264d-4812-8446-7ce83590a9b7-kube-api-access-f2gc7" pod "7804a436-264d-4812-8446-7ce83590a9b7" (UID: "7804a436-264d-4812-8446-7ce83590a9b7"). InnerVolumeSpecName "kube-api-access-f2gc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:02:40.937059 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:40.937036    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7804a436-264d-4812-8446-7ce83590a9b7-maas-api-xks-ca" pod "7804a436-264d-4812-8446-7ce83590a9b7" (UID: "7804a436-264d-4812-8446-7ce83590a9b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:02:40.962207 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:40.962172    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6447008-d4gbk"
Oct 30 05:02:41.036764 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:41.036691    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2gc7\" (UniqueName: \"kubernetes.io/projected/7804a436-264d-4812-8446-7ce83590a9b7-kube-api-access-f2gc7\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:02:41.036764 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:41.036725    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7804a436-264d-4812-8446-7ce83590a9b7-maas-api-openshift-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:02:41.036764 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:41.036740    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7804a436-264d-4812-8446-7ce83590a9b7-maas-api-xks-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:02:41.258433 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:02:41.258389    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a171566_49dd_491e_8f7b_a07f07c15453.slice/crio-71f77ef3b07cfe702898bcc481c3df4be74582bb030fe2ded908ea3a42a3c722 WatchSource:0}: Error finding container 71f77ef3b07cfe702898bcc481c3df4be74582bb030fe2ded908ea3a42a3c722: Status 404 returned error can't find the container with id 71f77ef3b07cfe702898bcc481c3df4be74582bb030fe2ded908ea3a42a3c722
Oct 30 05:02:41.259633 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:41.259600    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6447008-d4gbk"]
Oct 30 05:02:41.628726 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:41.628701    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:02:41.646166 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:41.646144    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8db1e0-t48w9"
Oct 30 05:02:41.646526 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:41.646144    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8db1e0-t48w9" event={"ID":"7804a436-264d-4812-8446-7ce83590a9b7","Type":"ContainerDied","Data":"760f7aa1332a96f12c5df550c1645663607bf8ac6482aa5c46f17071b0aa8bd8"}
Oct 30 05:02:41.646526 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:41.646289    5898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="760f7aa1332a96f12c5df550c1645663607bf8ac6482aa5c46f17071b0aa8bd8"
Oct 30 05:02:41.647930 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:41.647892    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6447008-d4gbk" event={"ID":"5a171566-49dd-491e-8f7b-a07f07c15453","Type":"ContainerStarted","Data":"0f5f4b3269fb66dce2acc3a7f61bf0f6f9fe1e2a57558aac9042ed428c13659f"}
Oct 30 05:02:41.648054 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:41.647950    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6447008-d4gbk" event={"ID":"5a171566-49dd-491e-8f7b-a07f07c15453","Type":"ContainerStarted","Data":"71f77ef3b07cfe702898bcc481c3df4be74582bb030fe2ded908ea3a42a3c722"}
Oct 30 05:02:41.689674 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:41.689618    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6447008-d4gbk" podStartSLOduration=1.689599081 podStartE2EDuration="1.689599081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:02:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:02:41.683720351 +0000 UTC m=+1090.907540253" watchObservedRunningTime="2026-10-30 05:02:41.689599081 +0000 UTC m=+1090.913418985"
Oct 30 05:02:42.656041 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:42.656003    5898 generic.go:356] "Generic (PLEG): container finished" podID="5a171566-49dd-491e-8f7b-a07f07c15453" containerID="0f5f4b3269fb66dce2acc3a7f61bf0f6f9fe1e2a57558aac9042ed428c13659f" exitCode=0
Oct 30 05:02:42.656461 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:42.656086    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6447008-d4gbk" event={"ID":"5a171566-49dd-491e-8f7b-a07f07c15453","Type":"ContainerDied","Data":"0f5f4b3269fb66dce2acc3a7f61bf0f6f9fe1e2a57558aac9042ed428c13659f"}
Oct 30 05:02:43.449115 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:43.449082    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:02:43.825812 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:43.825787    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6447008-d4gbk"
Oct 30 05:02:43.975797 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:43.975711    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a171566-49dd-491e-8f7b-a07f07c15453-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a171566-49dd-491e-8f7b-a07f07c15453-maas-api-openshift-ca\") pod \"5a171566-49dd-491e-8f7b-a07f07c15453\" (UID: \"5a171566-49dd-491e-8f7b-a07f07c15453\") "
Oct 30 05:02:43.975987 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:43.975804    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a171566-49dd-491e-8f7b-a07f07c15453-kube-api-access-g75zg\" (UniqueName: \"kubernetes.io/projected/5a171566-49dd-491e-8f7b-a07f07c15453-kube-api-access-g75zg\") pod \"5a171566-49dd-491e-8f7b-a07f07c15453\" (UID: \"5a171566-49dd-491e-8f7b-a07f07c15453\") "
Oct 30 05:02:43.975987 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:43.975871    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a171566-49dd-491e-8f7b-a07f07c15453-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a171566-49dd-491e-8f7b-a07f07c15453-maas-api-xks-ca\") pod \"5a171566-49dd-491e-8f7b-a07f07c15453\" (UID: \"5a171566-49dd-491e-8f7b-a07f07c15453\") "
Oct 30 05:02:43.976156 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:43.976132    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a171566-49dd-491e-8f7b-a07f07c15453-maas-api-openshift-ca" pod "5a171566-49dd-491e-8f7b-a07f07c15453" (UID: "5a171566-49dd-491e-8f7b-a07f07c15453"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:02:43.976454 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:43.976435    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a171566-49dd-491e-8f7b-a07f07c15453-maas-api-openshift-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:02:43.978009 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:43.977987    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a171566-49dd-491e-8f7b-a07f07c15453-maas-api-xks-ca" pod "5a171566-49dd-491e-8f7b-a07f07c15453" (UID: "5a171566-49dd-491e-8f7b-a07f07c15453"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:02:43.978099 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:43.978032    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a171566-49dd-491e-8f7b-a07f07c15453-kube-api-access-g75zg" pod "5a171566-49dd-491e-8f7b-a07f07c15453" (UID: "5a171566-49dd-491e-8f7b-a07f07c15453"). InnerVolumeSpecName "kube-api-access-g75zg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:02:44.078078 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:44.078047    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g75zg\" (UniqueName: \"kubernetes.io/projected/5a171566-49dd-491e-8f7b-a07f07c15453-kube-api-access-g75zg\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:02:44.078078 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:44.078074    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a171566-49dd-491e-8f7b-a07f07c15453-maas-api-xks-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:02:44.672396 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:44.672365    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6447008-d4gbk"
Oct 30 05:02:44.672396 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:44.672383    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6447008-d4gbk" event={"ID":"5a171566-49dd-491e-8f7b-a07f07c15453","Type":"ContainerDied","Data":"71f77ef3b07cfe702898bcc481c3df4be74582bb030fe2ded908ea3a42a3c722"}
Oct 30 05:02:44.672614 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:44.672428    5898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71f77ef3b07cfe702898bcc481c3df4be74582bb030fe2ded908ea3a42a3c722"
Oct 30 05:02:45.139817 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:45.139782    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:02:46.921030 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:46.920997    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:02:47.050784 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:47.050745    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:02:47.399862 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:47.399830    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bdaa2cf-5k489"]
Oct 30 05:02:47.400440 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:47.400423    5898 cpu_manager.go:414] "containerMap: removing container" podUID="5a171566-49dd-491e-8f7b-a07f07c15453" containerName="revoke-keys"
Oct 30 05:02:47.400505 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:47.400442    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a171566-49dd-491e-8f7b-a07f07c15453" containerName="revoke-keys"
Oct 30 05:02:47.400505 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:47.400468    5898 cpu_manager.go:414] "containerMap: removing container" podUID="7804a436-264d-4812-8446-7ce83590a9b7" containerName="revoke-keys"
Oct 30 05:02:47.400505 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:47.400474    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7804a436-264d-4812-8446-7ce83590a9b7" containerName="revoke-keys"
Oct 30 05:02:47.400590 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:47.400573    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="7804a436-264d-4812-8446-7ce83590a9b7" containerName="revoke-keys"
Oct 30 05:02:47.400590 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:47.400586    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a171566-49dd-491e-8f7b-a07f07c15453" containerName="revoke-keys"
Oct 30 05:02:47.404269 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:47.404250    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bdaa2cf-5k489"
Oct 30 05:02:47.421695 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:47.421666    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bdaa2cf-5k489"]
Oct 30 05:02:47.428383 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:47.428355    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:02:47.428506 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:47.428494    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\""
Oct 30 05:02:47.524624 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:47.524581    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md2t9\" (UniqueName: \"kubernetes.io/projected/55c4c3a3-a899-4431-8a8b-44382fec5062-kube-api-access-md2t9\") pod \"maas-api-revoke-sub-7bdaa2cf-5k489\" (UID: \"55c4c3a3-a899-4431-8a8b-44382fec5062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bdaa2cf-5k489"
Oct 30 05:02:47.524782 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:47.524693    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55c4c3a3-a899-4431-8a8b-44382fec5062-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7bdaa2cf-5k489\" (UID: \"55c4c3a3-a899-4431-8a8b-44382fec5062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bdaa2cf-5k489"
Oct 30 05:02:47.524782 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:47.524767    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55c4c3a3-a899-4431-8a8b-44382fec5062-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7bdaa2cf-5k489\" (UID: \"55c4c3a3-a899-4431-8a8b-44382fec5062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bdaa2cf-5k489"
Oct 30 05:02:47.626148 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:47.626114    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md2t9\" (UniqueName: \"kubernetes.io/projected/55c4c3a3-a899-4431-8a8b-44382fec5062-kube-api-access-md2t9\") pod \"maas-api-revoke-sub-7bdaa2cf-5k489\" (UID: \"55c4c3a3-a899-4431-8a8b-44382fec5062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bdaa2cf-5k489"
Oct 30 05:02:47.626297 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:47.626167    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55c4c3a3-a899-4431-8a8b-44382fec5062-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7bdaa2cf-5k489\" (UID: \"55c4c3a3-a899-4431-8a8b-44382fec5062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bdaa2cf-5k489"
Oct 30 05:02:47.626297 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:47.626208    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55c4c3a3-a899-4431-8a8b-44382fec5062-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7bdaa2cf-5k489\" (UID: \"55c4c3a3-a899-4431-8a8b-44382fec5062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bdaa2cf-5k489"
Oct 30 05:02:47.626991 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:47.626960    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55c4c3a3-a899-4431-8a8b-44382fec5062-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7bdaa2cf-5k489\" (UID: \"55c4c3a3-a899-4431-8a8b-44382fec5062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bdaa2cf-5k489"
Oct 30 05:02:47.628568 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:47.628540    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55c4c3a3-a899-4431-8a8b-44382fec5062-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7bdaa2cf-5k489\" (UID: \"55c4c3a3-a899-4431-8a8b-44382fec5062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bdaa2cf-5k489"
Oct 30 05:02:47.644344 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:47.644321    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-md2t9\" (UniqueName: \"kubernetes.io/projected/55c4c3a3-a899-4431-8a8b-44382fec5062-kube-api-access-md2t9\") pod \"maas-api-revoke-sub-7bdaa2cf-5k489\" (UID: \"55c4c3a3-a899-4431-8a8b-44382fec5062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bdaa2cf-5k489"
Oct 30 05:02:47.719911 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:47.719817    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bdaa2cf-5k489"
Oct 30 05:02:48.049576 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:02:48.049527    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55c4c3a3_a899_4431_8a8b_44382fec5062.slice/crio-9e02fa2f44affc88930d4ff0bf2c7c147fe40528d898dd5f6ec0c3f5cac89244 WatchSource:0}: Error finding container 9e02fa2f44affc88930d4ff0bf2c7c147fe40528d898dd5f6ec0c3f5cac89244: Status 404 returned error can't find the container with id 9e02fa2f44affc88930d4ff0bf2c7c147fe40528d898dd5f6ec0c3f5cac89244
Oct 30 05:02:48.061660 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:48.061607    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bdaa2cf-5k489"]
Oct 30 05:02:48.716924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:48.714218    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bdaa2cf-5k489" event={"ID":"55c4c3a3-a899-4431-8a8b-44382fec5062","Type":"ContainerStarted","Data":"a5f08f41bd74d310ccf7ddbff896b60038a7e9953716f1efec072c890c0ec115"}
Oct 30 05:02:48.716924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:48.714289    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bdaa2cf-5k489" event={"ID":"55c4c3a3-a899-4431-8a8b-44382fec5062","Type":"ContainerStarted","Data":"9e02fa2f44affc88930d4ff0bf2c7c147fe40528d898dd5f6ec0c3f5cac89244"}
Oct 30 05:02:48.790931 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:48.790022    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bdaa2cf-5k489" podStartSLOduration=1.789999304 podStartE2EDuration="1.789999304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:02:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:02:48.771386422 +0000 UTC m=+1097.995206348" watchObservedRunningTime="2026-10-30 05:02:48.789999304 +0000 UTC m=+1098.013819208"
Oct 30 05:02:49.594935 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:49.591652    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:02:49.696981 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:49.684092    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5" podUID="ebf8f4a2-8ac6-48ec-bdba-3f6a99653264" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:49.696981 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:49.689992    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb" podUID="6f16bd6a-a7f0-49bb-844f-57b777bde2ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 05:02:49.776925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:49.770686    5898 generic.go:356] "Generic (PLEG): container finished" podID="55c4c3a3-a899-4431-8a8b-44382fec5062" containerID="a5f08f41bd74d310ccf7ddbff896b60038a7e9953716f1efec072c890c0ec115" exitCode=0
Oct 30 05:02:49.776925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:49.770940    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bdaa2cf-5k489" event={"ID":"55c4c3a3-a899-4431-8a8b-44382fec5062","Type":"ContainerDied","Data":"a5f08f41bd74d310ccf7ddbff896b60038a7e9953716f1efec072c890c0ec115"}
Oct 30 05:02:49.808929 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:49.807996    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49" podUID="05eef4eb-f379-4dc7-a947-123bd85a4968" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:49.883924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:49.882979    5898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g" podUID="70294a85-1050-4632-b3bf-c3e44a4c0b41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": context deadline exceeded"
Oct 30 05:02:49.944048 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:49.943999    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g" podUID="70294a85-1050-4632-b3bf-c3e44a4c0b41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 05:02:49.982322 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:49.979993    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6" podUID="b09804e3-8135-4f6d-ae34-c45d6da9728e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:50.039134 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:50.039087    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58" podUID="c27f3b21-1447-4af9-8d99-c04fa9b23cda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:50.086080 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:50.048042    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247" podUID="1d865d3e-a6e9-407e-844e-cc75fc7f1354" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:50.155992 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:50.136061    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9" podUID="1647f909-5580-4219-a2e1-30b4cd994379" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:50.155992 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:50.150141    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6" podUID="47ff272a-ac5c-4f59-bf7b-5c0692d09386" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:50.194422 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:50.194028    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh" podUID="38428a53-04c7-400a-9c78-52407e982478" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:50.387925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:50.380659    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849" podUID="a9707f50-d5e1-409f-b8f3-c58629677aaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:50.711985 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:50.704986    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx" podUID="d523008f-c740-4a41-89cb-e82e1ffa0094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 05:02:50.711985 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:50.706930    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb" podUID="6f16bd6a-a7f0-49bb-844f-57b777bde2ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:50.728017 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:50.727969    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5" podUID="ebf8f4a2-8ac6-48ec-bdba-3f6a99653264" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:50.761927 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:50.728038    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb"
Oct 30 05:02:50.762243 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:50.762215    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5"
Oct 30 05:02:50.836599 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:50.836544    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49" podUID="05eef4eb-f379-4dc7-a947-123bd85a4968" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 05:02:50.837275 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:50.837249    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49"
Oct 30 05:02:50.990731 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:50.988972    5898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6" podUID="b09804e3-8135-4f6d-ae34-c45d6da9728e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": context deadline exceeded"
Oct 30 05:02:50.999924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:50.998993    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6" podUID="b09804e3-8135-4f6d-ae34-c45d6da9728e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:50.999924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:50.999378    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g" podUID="70294a85-1050-4632-b3bf-c3e44a4c0b41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 05:02:51.138944 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:51.127237    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58" podUID="c27f3b21-1447-4af9-8d99-c04fa9b23cda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:51.158669 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:51.158619    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6" podUID="47ff272a-ac5c-4f59-bf7b-5c0692d09386" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Oct 30 05:02:51.299578 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:51.299483    5898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49" podUID="05eef4eb-f379-4dc7-a947-123bd85a4968" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": context deadline exceeded"
Oct 30 05:02:51.373761 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:51.341989    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9" podUID="1647f909-5580-4219-a2e1-30b4cd994379" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 05:02:51.396521 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:51.396474    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849" podUID="a9707f50-d5e1-409f-b8f3-c58629677aaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 05:02:51.399097 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:51.399060    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh" podUID="38428a53-04c7-400a-9c78-52407e982478" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 30 05:02:51.448094 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:51.448030    5898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6" podUID="47ff272a-ac5c-4f59-bf7b-5c0692d09386" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": context deadline exceeded"
Oct 30 05:02:51.489382 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:51.449363    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247" podUID="1d865d3e-a6e9-407e-844e-cc75fc7f1354" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 05:02:51.892862 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:51.892818    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49" podUID="05eef4eb-f379-4dc7-a947-123bd85a4968" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 05:02:51.992158 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:51.928062    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb" podUID="6f16bd6a-a7f0-49bb-844f-57b777bde2ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:51.994008 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:51.954095    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx" podUID="d523008f-c740-4a41-89cb-e82e1ffa0094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 05:02:52.051588 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:52.051539    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5" podUID="ebf8f4a2-8ac6-48ec-bdba-3f6a99653264" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 05:02:52.208979 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:52.147116    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6" podUID="b09804e3-8135-4f6d-ae34-c45d6da9728e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 05:02:52.289396 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:52.159009    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58" podUID="c27f3b21-1447-4af9-8d99-c04fa9b23cda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:52.341927 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:52.159345    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g" podUID="70294a85-1050-4632-b3bf-c3e44a4c0b41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 05:02:52.346278 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:52.257990    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6" podUID="47ff272a-ac5c-4f59-bf7b-5c0692d09386" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Oct 30 05:02:52.374548 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:52.374511    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9" podUID="1647f909-5580-4219-a2e1-30b4cd994379" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:52.419464 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:52.419422    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849" podUID="a9707f50-d5e1-409f-b8f3-c58629677aaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 05:02:52.422434 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:52.422401    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh" podUID="38428a53-04c7-400a-9c78-52407e982478" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:52.457926 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:02:52.450156    5898 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.435s"
Oct 30 05:02:52.498412 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:52.498333    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6"
Oct 30 05:02:52.502103 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:52.502074    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g"
Oct 30 05:02:52.527427 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:52.527395    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58"
Oct 30 05:02:52.567501 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:52.567471    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6"
Oct 30 05:02:52.567800 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:52.567780    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9"
Oct 30 05:02:52.567988 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:52.567973    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849"
Oct 30 05:02:52.606927 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:52.602863    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh"
Oct 30 05:02:52.657422 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:52.491295    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247" podUID="1d865d3e-a6e9-407e-844e-cc75fc7f1354" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 05:02:52.680382 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:52.680353    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247"
Oct 30 05:02:52.680676 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:52.680661    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx"
Oct 30 05:02:53.002599 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:53.002554    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx" podUID="d523008f-c740-4a41-89cb-e82e1ffa0094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:53.003510 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:53.003473    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb" podUID="6f16bd6a-a7f0-49bb-844f-57b777bde2ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:53.004401 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:53.004375    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49" podUID="05eef4eb-f379-4dc7-a947-123bd85a4968" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:53.072961 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:53.052033    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5" podUID="ebf8f4a2-8ac6-48ec-bdba-3f6a99653264" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:53.361004 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:53.276869    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9"
Oct 30 05:02:53.361004 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:53.276980    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx"
Oct 30 05:02:53.361004 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:53.297291    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6" podUID="b09804e3-8135-4f6d-ae34-c45d6da9728e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:53.361004 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:53.321260    5898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247" podUID="1d865d3e-a6e9-407e-844e-cc75fc7f1354" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:53.361004 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:53.324051    5898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh" podUID="38428a53-04c7-400a-9c78-52407e982478" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": context deadline exceeded"
Oct 30 05:02:53.361004 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:53.324125    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58" podUID="c27f3b21-1447-4af9-8d99-c04fa9b23cda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 05:02:53.361004 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:53.347144    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g" podUID="70294a85-1050-4632-b3bf-c3e44a4c0b41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:53.374330 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:53.369145    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bdaa2cf-5k489"
Oct 30 05:02:53.374330 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:53.372778    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6" podUID="47ff272a-ac5c-4f59-bf7b-5c0692d09386" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:53.422048 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:53.421997    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849" podUID="a9707f50-d5e1-409f-b8f3-c58629677aaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:53.423571 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:53.423540    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh" podUID="38428a53-04c7-400a-9c78-52407e982478" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:53.503951 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:53.482978    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55c4c3a3-a899-4431-8a8b-44382fec5062-kube-api-access-md2t9\" (UniqueName: \"kubernetes.io/projected/55c4c3a3-a899-4431-8a8b-44382fec5062-kube-api-access-md2t9\") pod \"55c4c3a3-a899-4431-8a8b-44382fec5062\" (UID: \"55c4c3a3-a899-4431-8a8b-44382fec5062\") "
Oct 30 05:02:53.503951 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:53.483023    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55c4c3a3-a899-4431-8a8b-44382fec5062-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55c4c3a3-a899-4431-8a8b-44382fec5062-maas-api-openshift-ca\") pod \"55c4c3a3-a899-4431-8a8b-44382fec5062\" (UID: \"55c4c3a3-a899-4431-8a8b-44382fec5062\") "
Oct 30 05:02:53.503951 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:53.483057    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55c4c3a3-a899-4431-8a8b-44382fec5062-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55c4c3a3-a899-4431-8a8b-44382fec5062-maas-api-xks-ca\") pod \"55c4c3a3-a899-4431-8a8b-44382fec5062\" (UID: \"55c4c3a3-a899-4431-8a8b-44382fec5062\") "
Oct 30 05:02:53.503951 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:53.495194    5898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb" podUID="6f16bd6a-a7f0-49bb-844f-57b777bde2ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:53.503951 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:53.495279    5898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx" podUID="d523008f-c740-4a41-89cb-e82e1ffa0094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:53.506354 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:53.506222    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55c4c3a3-a899-4431-8a8b-44382fec5062-maas-api-openshift-ca" pod "55c4c3a3-a899-4431-8a8b-44382fec5062" (UID: "55c4c3a3-a899-4431-8a8b-44382fec5062"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:02:53.559990 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:53.559952    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55c4c3a3-a899-4431-8a8b-44382fec5062-maas-api-xks-ca" pod "55c4c3a3-a899-4431-8a8b-44382fec5062" (UID: "55c4c3a3-a899-4431-8a8b-44382fec5062"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:02:53.577497 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:53.577455    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55c4c3a3-a899-4431-8a8b-44382fec5062-kube-api-access-md2t9" pod "55c4c3a3-a899-4431-8a8b-44382fec5062" (UID: "55c4c3a3-a899-4431-8a8b-44382fec5062"). InnerVolumeSpecName "kube-api-access-md2t9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:02:53.590238 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:53.590136    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-md2t9\" (UniqueName: \"kubernetes.io/projected/55c4c3a3-a899-4431-8a8b-44382fec5062-kube-api-access-md2t9\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:02:53.590238 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:53.590178    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55c4c3a3-a899-4431-8a8b-44382fec5062-maas-api-openshift-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:02:53.590238 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:53.590195    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55c4c3a3-a899-4431-8a8b-44382fec5062-maas-api-xks-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:02:53.682389 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:53.681233    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247" podUID="1d865d3e-a6e9-407e-844e-cc75fc7f1354" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 05:02:53.726862 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:53.726831    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bdaa2cf-5k489"
Oct 30 05:02:53.742110 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:53.742062    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bdaa2cf-5k489" event={"ID":"55c4c3a3-a899-4431-8a8b-44382fec5062","Type":"ContainerDied","Data":"9e02fa2f44affc88930d4ff0bf2c7c147fe40528d898dd5f6ec0c3f5cac89244"}
Oct 30 05:02:53.742327 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:53.742152    5898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e02fa2f44affc88930d4ff0bf2c7c147fe40528d898dd5f6ec0c3f5cac89244"
Oct 30 05:02:54.005257 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:54.005154    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb" podUID="6f16bd6a-a7f0-49bb-844f-57b777bde2ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:54.038342 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:54.038283    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49" podUID="05eef4eb-f379-4dc7-a947-123bd85a4968" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:54.052028 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:54.051986    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9" podUID="1647f909-5580-4219-a2e1-30b4cd994379" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:54.052465 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:54.052250    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5" podUID="ebf8f4a2-8ac6-48ec-bdba-3f6a99653264" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:54.253025 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:54.252963    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8867-cde874"]
Oct 30 05:02:54.278978 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:54.257739    5898 cpu_manager.go:414] "containerMap: removing container" podUID="55c4c3a3-a899-4431-8a8b-44382fec5062" containerName="revoke-keys"
Oct 30 05:02:54.279219 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:54.279195    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55c4c3a3-a899-4431-8a8b-44382fec5062" containerName="revoke-keys"
Oct 30 05:02:54.279582 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:54.279564    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="55c4c3a3-a899-4431-8a8b-44382fec5062" containerName="revoke-keys"
Oct 30 05:02:54.290534 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:54.289147    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8867-cde874"
Oct 30 05:02:54.318401 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:54.300892    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6" podUID="b09804e3-8135-4f6d-ae34-c45d6da9728e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:54.349930 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:54.346047    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58" podUID="c27f3b21-1447-4af9-8d99-c04fa9b23cda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:54.369844 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:54.364396    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g" podUID="70294a85-1050-4632-b3bf-c3e44a4c0b41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:54.380159 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:54.380114    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6" podUID="47ff272a-ac5c-4f59-bf7b-5c0692d09386" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Oct 30 05:02:54.388130 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:54.385499    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-twdr8\""
Oct 30 05:02:54.429056 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:54.429010    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh" podUID="38428a53-04c7-400a-9c78-52407e982478" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:54.430987 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:54.429369    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849" podUID="a9707f50-d5e1-409f-b8f3-c58629677aaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 05:02:54.473379 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:54.473340    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8867-cde874"]
Oct 30 05:02:54.482004 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:54.478352    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8rm9\" (UniqueName: \"kubernetes.io/projected/4cca6cfa-12f9-450a-8fb3-2e3050539a5c-kube-api-access-d8rm9\") pod \"test-curl-8867-cde874\" (UID: \"4cca6cfa-12f9-450a-8fb3-2e3050539a5c\") " pod="opendatahub/test-curl-8867-cde874"
Oct 30 05:02:54.595206 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:54.595169    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8rm9\" (UniqueName: \"kubernetes.io/projected/4cca6cfa-12f9-450a-8fb3-2e3050539a5c-kube-api-access-d8rm9\") pod \"test-curl-8867-cde874\" (UID: \"4cca6cfa-12f9-450a-8fb3-2e3050539a5c\") " pod="opendatahub/test-curl-8867-cde874"
Oct 30 05:02:54.700996 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:54.700946    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247" podUID="1d865d3e-a6e9-407e-844e-cc75fc7f1354" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:54.724927 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:54.720858    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8rm9\" (UniqueName: \"kubernetes.io/projected/4cca6cfa-12f9-450a-8fb3-2e3050539a5c-kube-api-access-d8rm9\") pod \"test-curl-8867-cde874\" (UID: \"4cca6cfa-12f9-450a-8fb3-2e3050539a5c\") " pod="opendatahub/test-curl-8867-cde874"
Oct 30 05:02:54.740440 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:54.740404    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8867-cde874"
Oct 30 05:02:54.845321 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:54.820594    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kz7km" podUID="6b75122d-eff3-404c-aaaa-524c014fa7fc" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.19:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:55.028926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:55.028378    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb" podUID="6f16bd6a-a7f0-49bb-844f-57b777bde2ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:55.049437 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:55.049392    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49" podUID="05eef4eb-f379-4dc7-a947-123bd85a4968" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 05:02:55.065511 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:55.065468    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5" podUID="ebf8f4a2-8ac6-48ec-bdba-3f6a99653264" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:55.066736 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:55.066705    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9" podUID="1647f909-5580-4219-a2e1-30b4cd994379" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 05:02:55.067352 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:55.067333    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9"
Oct 30 05:02:55.272080 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:55.271985    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx" podUID="d523008f-c740-4a41-89cb-e82e1ffa0094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:55.301742 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:55.301700    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6" podUID="b09804e3-8135-4f6d-ae34-c45d6da9728e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 05:02:55.391619 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:55.346537    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58" podUID="c27f3b21-1447-4af9-8d99-c04fa9b23cda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 05:02:55.391619 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:55.369846    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g" podUID="70294a85-1050-4632-b3bf-c3e44a4c0b41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:55.391619 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:55.386996    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6" podUID="47ff272a-ac5c-4f59-bf7b-5c0692d09386" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:55.430304 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:55.430255    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh" podUID="38428a53-04c7-400a-9c78-52407e982478" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:55.576004 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:55.574017    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849" podUID="a9707f50-d5e1-409f-b8f3-c58629677aaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:55.718171 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:55.705635    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247" podUID="1d865d3e-a6e9-407e-844e-cc75fc7f1354" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:55.957605 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:55.949765    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8867-cde874"]
Oct 30 05:02:56.034177 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:56.033679    5898 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 05:02:56.048149 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:56.037074    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb" podUID="6f16bd6a-a7f0-49bb-844f-57b777bde2ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:56.052675 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:56.052632    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49" podUID="05eef4eb-f379-4dc7-a947-123bd85a4968" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:56.066022 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:56.065985    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5" podUID="ebf8f4a2-8ac6-48ec-bdba-3f6a99653264" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:56.067125 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:56.067094    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9" podUID="1647f909-5580-4219-a2e1-30b4cd994379" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:56.286552 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:56.286431    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx" podUID="d523008f-c740-4a41-89cb-e82e1ffa0094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:56.287971 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:56.286965    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx"
Oct 30 05:02:56.420966 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:56.380101    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g" podUID="70294a85-1050-4632-b3bf-c3e44a4c0b41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:56.454970 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:56.434802    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh" podUID="38428a53-04c7-400a-9c78-52407e982478" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 30 05:02:56.454970 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:56.436013    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6" podUID="47ff272a-ac5c-4f59-bf7b-5c0692d09386" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Oct 30 05:02:56.490791 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:56.489068    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58" podUID="c27f3b21-1447-4af9-8d99-c04fa9b23cda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:56.490791 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:56.489450    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6" podUID="b09804e3-8135-4f6d-ae34-c45d6da9728e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:56.659010 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:56.590381    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849" podUID="a9707f50-d5e1-409f-b8f3-c58629677aaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:56.751925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:56.722558    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247" podUID="1d865d3e-a6e9-407e-844e-cc75fc7f1354" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:56.972299 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:02:56.970442    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:02:57.002205 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:57.002149    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9"
Oct 30 05:02:57.044937 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:57.036058    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8867-cde874" event={"ID":"4cca6cfa-12f9-450a-8fb3-2e3050539a5c","Type":"ContainerStarted","Data":"32504cc7a8f9c11e81f3d72093cf8687d50872cbc4aa9436c29bb7e5713aba38"}
Oct 30 05:02:57.044937 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:57.044358    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb" podUID="6f16bd6a-a7f0-49bb-844f-57b777bde2ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:57.045495 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:57.045060    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6"
Oct 30 05:02:57.045989 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:57.045969    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx"
Oct 30 05:02:57.060353 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:57.060319    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49" podUID="05eef4eb-f379-4dc7-a947-123bd85a4968" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:57.115701 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:57.115463    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849"
Oct 30 05:02:57.157961 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:57.157936    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247"
Oct 30 05:02:57.162920 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:57.160862    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58"
Oct 30 05:02:57.232928 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:57.232819    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g"
Oct 30 05:02:57.236439 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:57.236310    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5" podUID="ebf8f4a2-8ac6-48ec-bdba-3f6a99653264" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:02:57.273132 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:57.273099    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5"
Oct 30 05:02:57.301091 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:57.301061    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49"
Oct 30 05:02:57.322975 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:57.322946    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh"
Oct 30 05:02:57.336400 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:57.336371    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb"
Oct 30 05:02:57.337329 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:57.337307    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6"
Oct 30 05:02:59.054112 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:59.054001    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8867-cde874" event={"ID":"4cca6cfa-12f9-450a-8fb3-2e3050539a5c","Type":"ContainerStarted","Data":"3e4a1ee1ba31a02d51d811e6628acb883223a1bf247e6092be07c781e24d68d5"}
Oct 30 05:02:59.090422 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:59.090368    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8867-cde874" podStartSLOduration=3.35276206 podStartE2EDuration="6.090353447s" totalImagesPullingTime="2.737591387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:02:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:02:59.089396699 +0000 UTC m=+1108.313216629" watchObservedRunningTime="2026-10-30 05:02:59.090353447 +0000 UTC m=+1108.314173347"
Oct 30 05:02:59.546048 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:59.546017    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:02:59.727257 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:59.727222    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50c66ea3-fkmqx"]
Oct 30 05:02:59.731447 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:59.731418    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50c66ea3-fkmqx"
Oct 30 05:02:59.748205 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:59.748173    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50c66ea3-fkmqx"]
Oct 30 05:02:59.754611 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:59.754569    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:02:59.754767 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:59.754612    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\""
Oct 30 05:02:59.900727 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:59.900689    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1d68ab4-b486-4ce1-b92e-e6caf2391dd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50c66ea3-fkmqx\" (UID: \"f1d68ab4-b486-4ce1-b92e-e6caf2391dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50c66ea3-fkmqx"
Oct 30 05:02:59.900928 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:59.900827    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1d68ab4-b486-4ce1-b92e-e6caf2391dd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50c66ea3-fkmqx\" (UID: \"f1d68ab4-b486-4ce1-b92e-e6caf2391dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50c66ea3-fkmqx"
Oct 30 05:02:59.900928 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:02:59.900877    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5tzv\" (UniqueName: \"kubernetes.io/projected/f1d68ab4-b486-4ce1-b92e-e6caf2391dd9-kube-api-access-b5tzv\") pod \"maas-api-revoke-sub-50c66ea3-fkmqx\" (UID: \"f1d68ab4-b486-4ce1-b92e-e6caf2391dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50c66ea3-fkmqx"
Oct 30 05:03:00.002637 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:00.002597    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1d68ab4-b486-4ce1-b92e-e6caf2391dd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50c66ea3-fkmqx\" (UID: \"f1d68ab4-b486-4ce1-b92e-e6caf2391dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50c66ea3-fkmqx"
Oct 30 05:03:00.002836 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:00.002656    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5tzv\" (UniqueName: \"kubernetes.io/projected/f1d68ab4-b486-4ce1-b92e-e6caf2391dd9-kube-api-access-b5tzv\") pod \"maas-api-revoke-sub-50c66ea3-fkmqx\" (UID: \"f1d68ab4-b486-4ce1-b92e-e6caf2391dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50c66ea3-fkmqx"
Oct 30 05:03:00.003137 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:00.003103    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1d68ab4-b486-4ce1-b92e-e6caf2391dd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50c66ea3-fkmqx\" (UID: \"f1d68ab4-b486-4ce1-b92e-e6caf2391dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50c66ea3-fkmqx"
Oct 30 05:03:00.003455 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:00.003428    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1d68ab4-b486-4ce1-b92e-e6caf2391dd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50c66ea3-fkmqx\" (UID: \"f1d68ab4-b486-4ce1-b92e-e6caf2391dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50c66ea3-fkmqx"
Oct 30 05:03:00.005418 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:00.005397    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1d68ab4-b486-4ce1-b92e-e6caf2391dd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50c66ea3-fkmqx\" (UID: \"f1d68ab4-b486-4ce1-b92e-e6caf2391dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50c66ea3-fkmqx"
Oct 30 05:03:00.024001 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:00.023970    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5tzv\" (UniqueName: \"kubernetes.io/projected/f1d68ab4-b486-4ce1-b92e-e6caf2391dd9-kube-api-access-b5tzv\") pod \"maas-api-revoke-sub-50c66ea3-fkmqx\" (UID: \"f1d68ab4-b486-4ce1-b92e-e6caf2391dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50c66ea3-fkmqx"
Oct 30 05:03:00.046880 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:00.046850    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50c66ea3-fkmqx"
Oct 30 05:03:00.118501 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:00.118453    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8867-cde874"]
Oct 30 05:03:00.123397 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:00.123366    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8867-cde874"]
Oct 30 05:03:00.283498 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:00.283462    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8867-539628"]
Oct 30 05:03:00.284105 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:00.284089    5898 cpu_manager.go:414] "containerMap: removing container" podUID="4cca6cfa-12f9-450a-8fb3-2e3050539a5c" containerName="test-curl-8867-cde874"
Oct 30 05:03:00.284165 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:00.284106    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cca6cfa-12f9-450a-8fb3-2e3050539a5c" containerName="test-curl-8867-cde874"
Oct 30 05:03:00.284279 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:00.284259    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cca6cfa-12f9-450a-8fb3-2e3050539a5c" containerName="test-curl-8867-cde874"
Oct 30 05:03:00.289053 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:00.289035    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8867-539628"
Oct 30 05:03:00.301078 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:00.301055    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8867-539628"]
Oct 30 05:03:00.375672 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:03:00.375637    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1d68ab4_b486_4ce1_b92e_e6caf2391dd9.slice/crio-c7c30129b697712a64b22c6d214dde9405ecd7b7e00ed2e1cb460b894812e9c5 WatchSource:0}: Error finding container c7c30129b697712a64b22c6d214dde9405ecd7b7e00ed2e1cb460b894812e9c5: Status 404 returned error can't find the container with id c7c30129b697712a64b22c6d214dde9405ecd7b7e00ed2e1cb460b894812e9c5
Oct 30 05:03:00.376223 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:00.376198    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50c66ea3-fkmqx"]
Oct 30 05:03:00.409239 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:00.409214    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgsf4\" (UniqueName: \"kubernetes.io/projected/0ac9be68-8672-4a08-8319-a8154e2e9193-kube-api-access-xgsf4\") pod \"test-curl-8867-539628\" (UID: \"0ac9be68-8672-4a08-8319-a8154e2e9193\") " pod="opendatahub/test-curl-8867-539628"
Oct 30 05:03:00.511008 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:00.510968    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgsf4\" (UniqueName: \"kubernetes.io/projected/0ac9be68-8672-4a08-8319-a8154e2e9193-kube-api-access-xgsf4\") pod \"test-curl-8867-539628\" (UID: \"0ac9be68-8672-4a08-8319-a8154e2e9193\") " pod="opendatahub/test-curl-8867-539628"
Oct 30 05:03:00.529796 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:00.529770    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgsf4\" (UniqueName: \"kubernetes.io/projected/0ac9be68-8672-4a08-8319-a8154e2e9193-kube-api-access-xgsf4\") pod \"test-curl-8867-539628\" (UID: \"0ac9be68-8672-4a08-8319-a8154e2e9193\") " pod="opendatahub/test-curl-8867-539628"
Oct 30 05:03:00.604700 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:00.604674    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8867-539628"
Oct 30 05:03:00.903391 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:03:00.903361    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0ac9be68_8672_4a08_8319_a8154e2e9193.slice/crio-0b424500a20fa5998e05b00cd7d16706aee771b0dbd17c38c6e1fe04e5d56f67 WatchSource:0}: Error finding container 0b424500a20fa5998e05b00cd7d16706aee771b0dbd17c38c6e1fe04e5d56f67: Status 404 returned error can't find the container with id 0b424500a20fa5998e05b00cd7d16706aee771b0dbd17c38c6e1fe04e5d56f67
Oct 30 05:03:00.903811 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:00.903790    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8867-539628"]
Oct 30 05:03:01.088554 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:01.088506    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50c66ea3-fkmqx" event={"ID":"f1d68ab4-b486-4ce1-b92e-e6caf2391dd9","Type":"ContainerStarted","Data":"9d210bd9f30d5060b51b5001c713bd60449aa63df62324c461b4f58759c515b2"}
Oct 30 05:03:01.088748 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:01.088568    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50c66ea3-fkmqx" event={"ID":"f1d68ab4-b486-4ce1-b92e-e6caf2391dd9","Type":"ContainerStarted","Data":"c7c30129b697712a64b22c6d214dde9405ecd7b7e00ed2e1cb460b894812e9c5"}
Oct 30 05:03:01.090257 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:01.090230    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8867-539628" event={"ID":"0ac9be68-8672-4a08-8319-a8154e2e9193","Type":"ContainerStarted","Data":"9feeb691c7112d74d153de417e9bf535b9b3586c5c34cf5364ed8a7ac9f4efae"}
Oct 30 05:03:01.090386 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:01.090266    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8867-539628" event={"ID":"0ac9be68-8672-4a08-8319-a8154e2e9193","Type":"ContainerStarted","Data":"0b424500a20fa5998e05b00cd7d16706aee771b0dbd17c38c6e1fe04e5d56f67"}
Oct 30 05:03:01.090386 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:01.090312    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8867-cde874" podUID="4cca6cfa-12f9-450a-8fb3-2e3050539a5c" containerName="test-curl-8867-cde874" containerID="cri-o://3e4a1ee1ba31a02d51d811e6628acb883223a1bf247e6092be07c781e24d68d5" gracePeriod=2
Oct 30 05:03:01.126379 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:01.126324    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50c66ea3-fkmqx" podStartSLOduration=2.126307679 podStartE2EDuration="2.126307679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:02:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:03:01.12408093 +0000 UTC m=+1110.347900845" watchObservedRunningTime="2026-10-30 05:03:01.126307679 +0000 UTC m=+1110.350127579"
Oct 30 05:03:01.161553 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:01.161455    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8867-539628" podStartSLOduration=1.161438945 podStartE2EDuration="1.161438945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:03:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:03:01.158044334 +0000 UTC m=+1110.381864233" watchObservedRunningTime="2026-10-30 05:03:01.161438945 +0000 UTC m=+1110.385258846"
Oct 30 05:03:01.455687 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:01.455598    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:02.001391 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:02.001337    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8867-539628"]
Oct 30 05:03:02.006392 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:02.006362    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8867-539628"]
Oct 30 05:03:02.099008 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:02.098971    5898 generic.go:356] "Generic (PLEG): container finished" podID="f1d68ab4-b486-4ce1-b92e-e6caf2391dd9" containerID="9d210bd9f30d5060b51b5001c713bd60449aa63df62324c461b4f58759c515b2" exitCode=0
Oct 30 05:03:02.099152 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:02.099058    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50c66ea3-fkmqx" event={"ID":"f1d68ab4-b486-4ce1-b92e-e6caf2391dd9","Type":"ContainerDied","Data":"9d210bd9f30d5060b51b5001c713bd60449aa63df62324c461b4f58759c515b2"}
Oct 30 05:03:02.672846 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:02.672806    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:02.775988 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:02.775957    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:02.879289 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:02.879259    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:03.106570 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:03.106541    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8867-539628" podUID="0ac9be68-8672-4a08-8319-a8154e2e9193" containerName="test-curl-8867-539628" containerID="cri-o://9feeb691c7112d74d153de417e9bf535b9b3586c5c34cf5364ed8a7ac9f4efae" gracePeriod=2
Oct 30 05:03:03.116819 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:03.116792    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:03.307717 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:03.307665    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:03.336537 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:03.336508    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50c66ea3-fkmqx"
Oct 30 05:03:03.344528 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:03.344461    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8867-cde874"
Oct 30 05:03:03.353222 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:03.353186    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1d68ab4-b486-4ce1-b92e-e6caf2391dd9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1d68ab4-b486-4ce1-b92e-e6caf2391dd9-maas-api-xks-ca\") pod \"f1d68ab4-b486-4ce1-b92e-e6caf2391dd9\" (UID: \"f1d68ab4-b486-4ce1-b92e-e6caf2391dd9\") "
Oct 30 05:03:03.353367 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:03.353239    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1d68ab4-b486-4ce1-b92e-e6caf2391dd9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1d68ab4-b486-4ce1-b92e-e6caf2391dd9-maas-api-openshift-ca\") pod \"f1d68ab4-b486-4ce1-b92e-e6caf2391dd9\" (UID: \"f1d68ab4-b486-4ce1-b92e-e6caf2391dd9\") "
Oct 30 05:03:03.353367 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:03.353266    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1d68ab4-b486-4ce1-b92e-e6caf2391dd9-kube-api-access-b5tzv\" (UniqueName: \"kubernetes.io/projected/f1d68ab4-b486-4ce1-b92e-e6caf2391dd9-kube-api-access-b5tzv\") pod \"f1d68ab4-b486-4ce1-b92e-e6caf2391dd9\" (UID: \"f1d68ab4-b486-4ce1-b92e-e6caf2391dd9\") "
Oct 30 05:03:03.353877 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:03.353850    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1d68ab4-b486-4ce1-b92e-e6caf2391dd9-maas-api-openshift-ca" pod "f1d68ab4-b486-4ce1-b92e-e6caf2391dd9" (UID: "f1d68ab4-b486-4ce1-b92e-e6caf2391dd9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:03:03.355655 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:03.355616    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1d68ab4-b486-4ce1-b92e-e6caf2391dd9-maas-api-xks-ca" pod "f1d68ab4-b486-4ce1-b92e-e6caf2391dd9" (UID: "f1d68ab4-b486-4ce1-b92e-e6caf2391dd9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:03:03.355761 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:03.355671    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1d68ab4-b486-4ce1-b92e-e6caf2391dd9-kube-api-access-b5tzv" pod "f1d68ab4-b486-4ce1-b92e-e6caf2391dd9" (UID: "f1d68ab4-b486-4ce1-b92e-e6caf2391dd9"). InnerVolumeSpecName "kube-api-access-b5tzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:03:03.398544 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:03.398506    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:03.454861 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:03.454825    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cca6cfa-12f9-450a-8fb3-2e3050539a5c-kube-api-access-d8rm9\" (UniqueName: \"kubernetes.io/projected/4cca6cfa-12f9-450a-8fb3-2e3050539a5c-kube-api-access-d8rm9\") pod \"4cca6cfa-12f9-450a-8fb3-2e3050539a5c\" (UID: \"4cca6cfa-12f9-450a-8fb3-2e3050539a5c\") "
Oct 30 05:03:03.455855 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:03.455825    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1d68ab4-b486-4ce1-b92e-e6caf2391dd9-maas-api-xks-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:03:03.456012 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:03.455997    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1d68ab4-b486-4ce1-b92e-e6caf2391dd9-maas-api-openshift-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:03:03.456110 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:03.456097    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5tzv\" (UniqueName: \"kubernetes.io/projected/f1d68ab4-b486-4ce1-b92e-e6caf2391dd9-kube-api-access-b5tzv\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:03:03.492924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:03.483192    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cca6cfa-12f9-450a-8fb3-2e3050539a5c-kube-api-access-d8rm9" pod "4cca6cfa-12f9-450a-8fb3-2e3050539a5c" (UID: "4cca6cfa-12f9-450a-8fb3-2e3050539a5c"). InnerVolumeSpecName "kube-api-access-d8rm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:03:03.561926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:03.561060    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8rm9\" (UniqueName: \"kubernetes.io/projected/4cca6cfa-12f9-450a-8fb3-2e3050539a5c-kube-api-access-d8rm9\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:03:04.114594 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:04.114561    5898 generic.go:356] "Generic (PLEG): container finished" podID="4cca6cfa-12f9-450a-8fb3-2e3050539a5c" containerID="3e4a1ee1ba31a02d51d811e6628acb883223a1bf247e6092be07c781e24d68d5" exitCode=137
Oct 30 05:03:04.115038 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:04.114611    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8867-cde874"
Oct 30 05:03:04.115038 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:04.114656    5898 scope.go:122] "RemoveContainer" containerID="3e4a1ee1ba31a02d51d811e6628acb883223a1bf247e6092be07c781e24d68d5"
Oct 30 05:03:04.116069 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:04.116051    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50c66ea3-fkmqx"
Oct 30 05:03:04.116148 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:04.116064    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50c66ea3-fkmqx" event={"ID":"f1d68ab4-b486-4ce1-b92e-e6caf2391dd9","Type":"ContainerDied","Data":"c7c30129b697712a64b22c6d214dde9405ecd7b7e00ed2e1cb460b894812e9c5"}
Oct 30 05:03:04.116148 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:04.116112    5898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7c30129b697712a64b22c6d214dde9405ecd7b7e00ed2e1cb460b894812e9c5"
Oct 30 05:03:04.127657 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:04.127627    5898 scope.go:122] "RemoveContainer" containerID="3e4a1ee1ba31a02d51d811e6628acb883223a1bf247e6092be07c781e24d68d5"
Oct 30 05:03:04.127950 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:03:04.127924    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e4a1ee1ba31a02d51d811e6628acb883223a1bf247e6092be07c781e24d68d5\": container with ID starting with 3e4a1ee1ba31a02d51d811e6628acb883223a1bf247e6092be07c781e24d68d5 not found: ID does not exist" containerID="3e4a1ee1ba31a02d51d811e6628acb883223a1bf247e6092be07c781e24d68d5"
Oct 30 05:03:04.128029 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:04.127960    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e4a1ee1ba31a02d51d811e6628acb883223a1bf247e6092be07c781e24d68d5"} err="failed to get container status \"3e4a1ee1ba31a02d51d811e6628acb883223a1bf247e6092be07c781e24d68d5\": rpc error: code = NotFound desc = could not find container \"3e4a1ee1ba31a02d51d811e6628acb883223a1bf247e6092be07c781e24d68d5\": container with ID starting with 3e4a1ee1ba31a02d51d811e6628acb883223a1bf247e6092be07c781e24d68d5 not found: ID does not exist"
Oct 30 05:03:04.137075 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:03:04.137047    5898 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8867-cde874\" is forbidden: User \"system:node:ip-10-0-135-98.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-98.ec2.internal' and this object" podUID="4cca6cfa-12f9-450a-8fb3-2e3050539a5c" pod="opendatahub/test-curl-8867-cde874"
Oct 30 05:03:04.163660 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:03:04.163618    5898 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8867-cde874\" is forbidden: User \"system:node:ip-10-0-135-98.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-98.ec2.internal' and this object" podUID="4cca6cfa-12f9-450a-8fb3-2e3050539a5c" pod="opendatahub/test-curl-8867-cde874"
Oct 30 05:03:04.893600 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:04.893563    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cca6cfa-12f9-450a-8fb3-2e3050539a5c" path="/var/lib/kubelet/pods/4cca6cfa-12f9-450a-8fb3-2e3050539a5c/volumes"
Oct 30 05:03:04.909110 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:04.909083    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:05.292154 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:05.292121    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8867-539628"
Oct 30 05:03:05.389174 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:05.389140    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ac9be68-8672-4a08-8319-a8154e2e9193-kube-api-access-xgsf4\" (UniqueName: \"kubernetes.io/projected/0ac9be68-8672-4a08-8319-a8154e2e9193-kube-api-access-xgsf4\") pod \"0ac9be68-8672-4a08-8319-a8154e2e9193\" (UID: \"0ac9be68-8672-4a08-8319-a8154e2e9193\") "
Oct 30 05:03:05.391813 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:05.391782    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ac9be68-8672-4a08-8319-a8154e2e9193-kube-api-access-xgsf4" pod "0ac9be68-8672-4a08-8319-a8154e2e9193" (UID: "0ac9be68-8672-4a08-8319-a8154e2e9193"). InnerVolumeSpecName "kube-api-access-xgsf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:03:05.493873 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:05.493765    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgsf4\" (UniqueName: \"kubernetes.io/projected/0ac9be68-8672-4a08-8319-a8154e2e9193-kube-api-access-xgsf4\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:03:06.134458 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:06.134420    5898 generic.go:356] "Generic (PLEG): container finished" podID="0ac9be68-8672-4a08-8319-a8154e2e9193" containerID="9feeb691c7112d74d153de417e9bf535b9b3586c5c34cf5364ed8a7ac9f4efae" exitCode=137
Oct 30 05:03:06.134625 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:06.134467    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8867-539628"
Oct 30 05:03:06.134625 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:06.134518    5898 scope.go:122] "RemoveContainer" containerID="9feeb691c7112d74d153de417e9bf535b9b3586c5c34cf5364ed8a7ac9f4efae"
Oct 30 05:03:06.144467 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:06.144449    5898 scope.go:122] "RemoveContainer" containerID="9feeb691c7112d74d153de417e9bf535b9b3586c5c34cf5364ed8a7ac9f4efae"
Oct 30 05:03:06.144713 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:03:06.144694    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9feeb691c7112d74d153de417e9bf535b9b3586c5c34cf5364ed8a7ac9f4efae\": container with ID starting with 9feeb691c7112d74d153de417e9bf535b9b3586c5c34cf5364ed8a7ac9f4efae not found: ID does not exist" containerID="9feeb691c7112d74d153de417e9bf535b9b3586c5c34cf5364ed8a7ac9f4efae"
Oct 30 05:03:06.144760 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:06.144722    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9feeb691c7112d74d153de417e9bf535b9b3586c5c34cf5364ed8a7ac9f4efae"} err="failed to get container status \"9feeb691c7112d74d153de417e9bf535b9b3586c5c34cf5364ed8a7ac9f4efae\": rpc error: code = NotFound desc = could not find container \"9feeb691c7112d74d153de417e9bf535b9b3586c5c34cf5364ed8a7ac9f4efae\": container with ID starting with 9feeb691c7112d74d153de417e9bf535b9b3586c5c34cf5364ed8a7ac9f4efae not found: ID does not exist"
Oct 30 05:03:06.155391 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:03:06.155366    5898 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8867-539628\" is forbidden: User \"system:node:ip-10-0-135-98.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-98.ec2.internal' and this object" podUID="0ac9be68-8672-4a08-8319-a8154e2e9193" pod="opendatahub/test-curl-8867-539628"
Oct 30 05:03:06.778585 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:06.778553    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:06.879112 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:06.879074    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:06.890323 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:06.890287    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ac9be68-8672-4a08-8319-a8154e2e9193" path="/var/lib/kubelet/pods/0ac9be68-8672-4a08-8319-a8154e2e9193/volumes"
Oct 30 05:03:06.983223 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:06.983175    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:08.560700 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:08.560663    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:08.657851 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:08.657814    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:08.700793 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:08.700765    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d67de78b-c65b577cb-q4q8b"]
Oct 30 05:03:08.701345 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:08.701330    5898 cpu_manager.go:414] "containerMap: removing container" podUID="f1d68ab4-b486-4ce1-b92e-e6caf2391dd9" containerName="revoke-keys"
Oct 30 05:03:08.701396 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:08.701347    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1d68ab4-b486-4ce1-b92e-e6caf2391dd9" containerName="revoke-keys"
Oct 30 05:03:08.701396 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:08.701363    5898 cpu_manager.go:414] "containerMap: removing container" podUID="0ac9be68-8672-4a08-8319-a8154e2e9193" containerName="test-curl-8867-539628"
Oct 30 05:03:08.701396 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:08.701369    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ac9be68-8672-4a08-8319-a8154e2e9193" containerName="test-curl-8867-539628"
Oct 30 05:03:08.701503 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:08.701461    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1d68ab4-b486-4ce1-b92e-e6caf2391dd9" containerName="revoke-keys"
Oct 30 05:03:08.701503 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:08.701469    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ac9be68-8672-4a08-8319-a8154e2e9193" containerName="test-curl-8867-539628"
Oct 30 05:03:08.706406 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:08.706390    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d67de78b-c65b577cb-q4q8b"
Oct 30 05:03:08.723024 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:08.723000    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d67de78b-c65b577cb-q4q8b"]
Oct 30 05:03:08.729669 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:08.729642    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-d67de78b\""
Oct 30 05:03:08.840535 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:08.840490    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dk9z\" (UniqueName: \"kubernetes.io/projected/36521c09-d623-48e2-801a-e3be3e2e25fa-kube-api-access-2dk9z\") pod \"payload-processing-e2e-ait-d67de78b-c65b577cb-q4q8b\" (UID: \"36521c09-d623-48e2-801a-e3be3e2e25fa\") " pod="openshift-ingress/payload-processing-e2e-ait-d67de78b-c65b577cb-q4q8b"
Oct 30 05:03:08.840739 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:08.840623    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36521c09-d623-48e2-801a-e3be3e2e25fa-plugins-config-volume\") pod \"payload-processing-e2e-ait-d67de78b-c65b577cb-q4q8b\" (UID: \"36521c09-d623-48e2-801a-e3be3e2e25fa\") " pod="openshift-ingress/payload-processing-e2e-ait-d67de78b-c65b577cb-q4q8b"
Oct 30 05:03:08.942119 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:08.942086    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dk9z\" (UniqueName: \"kubernetes.io/projected/36521c09-d623-48e2-801a-e3be3e2e25fa-kube-api-access-2dk9z\") pod \"payload-processing-e2e-ait-d67de78b-c65b577cb-q4q8b\" (UID: \"36521c09-d623-48e2-801a-e3be3e2e25fa\") " pod="openshift-ingress/payload-processing-e2e-ait-d67de78b-c65b577cb-q4q8b"
Oct 30 05:03:08.942286 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:08.942179    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36521c09-d623-48e2-801a-e3be3e2e25fa-plugins-config-volume\") pod \"payload-processing-e2e-ait-d67de78b-c65b577cb-q4q8b\" (UID: \"36521c09-d623-48e2-801a-e3be3e2e25fa\") " pod="openshift-ingress/payload-processing-e2e-ait-d67de78b-c65b577cb-q4q8b"
Oct 30 05:03:08.942944 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:08.942922    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36521c09-d623-48e2-801a-e3be3e2e25fa-plugins-config-volume\") pod \"payload-processing-e2e-ait-d67de78b-c65b577cb-q4q8b\" (UID: \"36521c09-d623-48e2-801a-e3be3e2e25fa\") " pod="openshift-ingress/payload-processing-e2e-ait-d67de78b-c65b577cb-q4q8b"
Oct 30 05:03:08.968586 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:08.968552    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dk9z\" (UniqueName: \"kubernetes.io/projected/36521c09-d623-48e2-801a-e3be3e2e25fa-kube-api-access-2dk9z\") pod \"payload-processing-e2e-ait-d67de78b-c65b577cb-q4q8b\" (UID: \"36521c09-d623-48e2-801a-e3be3e2e25fa\") " pod="openshift-ingress/payload-processing-e2e-ait-d67de78b-c65b577cb-q4q8b"
Oct 30 05:03:09.020342 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:09.020312    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d67de78b-c65b577cb-q4q8b"
Oct 30 05:03:09.333508 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:09.333476    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d67de78b-c65b577cb-q4q8b"]
Oct 30 05:03:09.333698 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:03:09.333677    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36521c09_d623_48e2_801a_e3be3e2e25fa.slice/crio-0dddedcd9afba6545e69ef5083603bed97d40ffef3c6e5f69eb20a1462a954fe WatchSource:0}: Error finding container 0dddedcd9afba6545e69ef5083603bed97d40ffef3c6e5f69eb20a1462a954fe: Status 404 returned error can't find the container with id 0dddedcd9afba6545e69ef5083603bed97d40ffef3c6e5f69eb20a1462a954fe
Oct 30 05:03:10.176337 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:10.176262    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d67de78b-c65b577cb-q4q8b" event={"ID":"36521c09-d623-48e2-801a-e3be3e2e25fa","Type":"ContainerStarted","Data":"da87e7b36567b4ec059b377c3433acbcd98fd5651305015d1d32bb8dfcb8a039"}
Oct 30 05:03:10.176836 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:10.176392    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-d67de78b-c65b577cb-q4q8b"
Oct 30 05:03:10.176836 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:10.176417    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d67de78b-c65b577cb-q4q8b" event={"ID":"36521c09-d623-48e2-801a-e3be3e2e25fa","Type":"ContainerStarted","Data":"0dddedcd9afba6545e69ef5083603bed97d40ffef3c6e5f69eb20a1462a954fe"}
Oct 30 05:03:10.216448 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:10.216400    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-d67de78b-c65b577cb-q4q8b" podStartSLOduration=2.21638645 podStartE2EDuration="2.21638645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:03:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:03:10.213075915 +0000 UTC m=+1119.436895816" watchObservedRunningTime="2026-10-30 05:03:10.21638645 +0000 UTC m=+1119.440206350"
Oct 30 05:03:10.221425 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:10.221400    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:10.306503 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:10.306454    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:10.404698 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:10.404664    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:10.547154 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:10.547062    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:11.032732 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:11.032701    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:11.149321 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:11.149277    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:12.137003 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:12.136966    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:14.077717 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:14.077678    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:14.178372 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:14.178343    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:14.325444 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:14.325411    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:14.773632 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:14.773601    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:14.872698 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:14.872668    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:14.972651 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:14.972616    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:15.681311 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:15.681260    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:15.790726 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:15.790678    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:15.885526 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:15.885493    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:15.906795 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:15.906764    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b0fba292-65679f777d-tkn74"]
Oct 30 05:03:15.911206 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:15.911187    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b0fba292-65679f777d-tkn74"
Oct 30 05:03:15.927531 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:15.927507    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b0fba292-65679f777d-tkn74"]
Oct 30 05:03:15.934917 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:15.934840    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-b0fba292\""
Oct 30 05:03:15.934917 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:15.934890    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-b0fba292-dockercfg-p84m8\""
Oct 30 05:03:16.096527 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:16.095579    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr7v2\" (UniqueName: \"kubernetes.io/projected/afe14932-62ea-4859-95eb-f030280ac1e6-kube-api-access-zr7v2\") pod \"payload-processing-e2e-ait-b0fba292-65679f777d-tkn74\" (UID: \"afe14932-62ea-4859-95eb-f030280ac1e6\") " pod="openshift-ingress/payload-processing-e2e-ait-b0fba292-65679f777d-tkn74"
Oct 30 05:03:16.096527 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:16.095875    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afe14932-62ea-4859-95eb-f030280ac1e6-plugins-config-volume\") pod \"payload-processing-e2e-ait-b0fba292-65679f777d-tkn74\" (UID: \"afe14932-62ea-4859-95eb-f030280ac1e6\") " pod="openshift-ingress/payload-processing-e2e-ait-b0fba292-65679f777d-tkn74"
Oct 30 05:03:16.196807 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:16.196719    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afe14932-62ea-4859-95eb-f030280ac1e6-plugins-config-volume\") pod \"payload-processing-e2e-ait-b0fba292-65679f777d-tkn74\" (UID: \"afe14932-62ea-4859-95eb-f030280ac1e6\") " pod="openshift-ingress/payload-processing-e2e-ait-b0fba292-65679f777d-tkn74"
Oct 30 05:03:16.196993 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:16.196892    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr7v2\" (UniqueName: \"kubernetes.io/projected/afe14932-62ea-4859-95eb-f030280ac1e6-kube-api-access-zr7v2\") pod \"payload-processing-e2e-ait-b0fba292-65679f777d-tkn74\" (UID: \"afe14932-62ea-4859-95eb-f030280ac1e6\") " pod="openshift-ingress/payload-processing-e2e-ait-b0fba292-65679f777d-tkn74"
Oct 30 05:03:16.197620 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:16.197592    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afe14932-62ea-4859-95eb-f030280ac1e6-plugins-config-volume\") pod \"payload-processing-e2e-ait-b0fba292-65679f777d-tkn74\" (UID: \"afe14932-62ea-4859-95eb-f030280ac1e6\") " pod="openshift-ingress/payload-processing-e2e-ait-b0fba292-65679f777d-tkn74"
Oct 30 05:03:16.215429 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:16.215392    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr7v2\" (UniqueName: \"kubernetes.io/projected/afe14932-62ea-4859-95eb-f030280ac1e6-kube-api-access-zr7v2\") pod \"payload-processing-e2e-ait-b0fba292-65679f777d-tkn74\" (UID: \"afe14932-62ea-4859-95eb-f030280ac1e6\") " pod="openshift-ingress/payload-processing-e2e-ait-b0fba292-65679f777d-tkn74"
Oct 30 05:03:16.225856 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:16.225814    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b0fba292-65679f777d-tkn74"
Oct 30 05:03:16.340921 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:16.340874    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:16.441681 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:16.441581    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:16.538054 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:16.538029    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b0fba292-65679f777d-tkn74"]
Oct 30 05:03:16.538158 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:03:16.538108    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafe14932_62ea_4859_95eb_f030280ac1e6.slice/crio-9b8b9163a9170d4626c7ddc5da0391ee821df080863d4a4e5a39398c50f4fa2a WatchSource:0}: Error finding container 9b8b9163a9170d4626c7ddc5da0391ee821df080863d4a4e5a39398c50f4fa2a: Status 404 returned error can't find the container with id 9b8b9163a9170d4626c7ddc5da0391ee821df080863d4a4e5a39398c50f4fa2a
Oct 30 05:03:16.539243 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:16.539216    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:17.245977 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:17.245936    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b0fba292-65679f777d-tkn74" event={"ID":"afe14932-62ea-4859-95eb-f030280ac1e6","Type":"ContainerStarted","Data":"b94100dba9aa5bbf8b6ee39d670d63c2f9b87d830b61d8b6192e99621bf335e1"}
Oct 30 05:03:17.246455 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:17.246004    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b0fba292-65679f777d-tkn74" event={"ID":"afe14932-62ea-4859-95eb-f030280ac1e6","Type":"ContainerStarted","Data":"9b8b9163a9170d4626c7ddc5da0391ee821df080863d4a4e5a39398c50f4fa2a"}
Oct 30 05:03:17.246455 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:17.246032    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-b0fba292-65679f777d-tkn74"
Oct 30 05:03:17.290200 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:17.290139    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-b0fba292-65679f777d-tkn74" podStartSLOduration=2.290125615 podStartE2EDuration="2.290125615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:03:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:03:17.287656487 +0000 UTC m=+1126.511476385" watchObservedRunningTime="2026-10-30 05:03:17.290125615 +0000 UTC m=+1126.513945515"
Oct 30 05:03:17.300868 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:17.300844    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:19.368926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:19.359008    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58" podUID="c27f3b21-1447-4af9-8d99-c04fa9b23cda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:03:19.380039 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:19.379990    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6" podUID="47ff272a-ac5c-4f59-bf7b-5c0692d09386" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:03:19.387036 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:19.386990    5898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9" podUID="1647f909-5580-4219-a2e1-30b4cd994379" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:03:19.437081 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:19.437027    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849" podUID="a9707f50-d5e1-409f-b8f3-c58629677aaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:03:19.532112 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:19.532052    5898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849" podUID="a9707f50-d5e1-409f-b8f3-c58629677aaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:03:19.956933 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:19.956106    5898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g" podUID="70294a85-1050-4632-b3bf-c3e44a4c0b41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:03:19.983076 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:19.983025    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9" podUID="1647f909-5580-4219-a2e1-30b4cd994379" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:03:20.002935 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:20.002875    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49" podUID="05eef4eb-f379-4dc7-a947-123bd85a4968" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:03:20.050921 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:20.050855    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx" podUID="d523008f-c740-4a41-89cb-e82e1ffa0094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 05:03:20.070924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:20.061981    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5" podUID="ebf8f4a2-8ac6-48ec-bdba-3f6a99653264" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:03:20.208982 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:20.208204    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247" podUID="1d865d3e-a6e9-407e-844e-cc75fc7f1354" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:03:20.311261 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:20.311178    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6" podUID="b09804e3-8135-4f6d-ae34-c45d6da9728e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:03:20.362997 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:20.361977    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g" podUID="70294a85-1050-4632-b3bf-c3e44a4c0b41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:03:20.362997 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:20.362957    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58" podUID="c27f3b21-1447-4af9-8d99-c04fa9b23cda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 05:03:20.363446 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:20.363421    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58"
Oct 30 05:03:20.384671 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:20.384628    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6" podUID="47ff272a-ac5c-4f59-bf7b-5c0692d09386" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:03:20.385716 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:20.385691    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6"
Oct 30 05:03:20.438924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:20.434084    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh" podUID="38428a53-04c7-400a-9c78-52407e982478" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 30 05:03:20.439294 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:20.439264    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849" podUID="a9707f50-d5e1-409f-b8f3-c58629677aaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:03:20.439866 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:20.439843    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849"
Oct 30 05:03:21.022932 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:20.989302    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9" podUID="1647f909-5580-4219-a2e1-30b4cd994379" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 05:03:21.022932 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:21.022382    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49" podUID="05eef4eb-f379-4dc7-a947-123bd85a4968" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:03:21.022932 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:21.022848    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb" podUID="6f16bd6a-a7f0-49bb-844f-57b777bde2ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:03:21.070932 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:21.062669    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5" podUID="ebf8f4a2-8ac6-48ec-bdba-3f6a99653264" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 05:03:21.269276 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:21.269237    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247" podUID="1d865d3e-a6e9-407e-844e-cc75fc7f1354" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:03:21.324198 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:21.324157    5898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6" podUID="47ff272a-ac5c-4f59-bf7b-5c0692d09386" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": context deadline exceeded"
Oct 30 05:03:21.324669 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:21.324644    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6" podUID="b09804e3-8135-4f6d-ae34-c45d6da9728e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 05:03:21.716980 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:21.699737    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6" podUID="47ff272a-ac5c-4f59-bf7b-5c0692d09386" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:03:21.745799 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:21.745755    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh" podUID="38428a53-04c7-400a-9c78-52407e982478" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:03:21.774916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:21.774855    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58" podUID="c27f3b21-1447-4af9-8d99-c04fa9b23cda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 05:03:21.824950 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:21.812618    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849" podUID="a9707f50-d5e1-409f-b8f3-c58629677aaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 05:03:21.930369 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:03:21.914639    5898 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.029s"
Oct 30 05:03:21.930369 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:21.914728    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9"
Oct 30 05:03:21.930369 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:21.914806    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49"
Oct 30 05:03:21.930369 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:21.914847    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5"
Oct 30 05:03:21.930369 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:21.914882    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247"
Oct 30 05:03:21.930369 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:21.914935    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6"
Oct 30 05:03:21.930369 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:21.914969    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh"
Oct 30 05:03:21.930369 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:21.915000    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9"
Oct 30 05:03:21.930369 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:21.915030    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5"
Oct 30 05:03:21.930369 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:21.915059    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247"
Oct 30 05:03:21.930369 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:21.915096    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-d67de78b-c65b577cb-q4q8b"
Oct 30 05:03:21.930369 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:21.915142    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6"
Oct 30 05:03:21.930369 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:21.915171    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49"
Oct 30 05:03:21.930369 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:21.915201    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh"
Oct 30 05:03:21.930369 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:21.927573    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849"
Oct 30 05:03:21.931244 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:21.930652    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58"
Oct 30 05:03:21.954146 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:21.954118    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6"
Oct 30 05:03:22.869434 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:22.869401    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:24.915208 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:24.915175    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36b03010-ss8rh"]
Oct 30 05:03:24.919468 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:24.919446    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b03010-ss8rh"
Oct 30 05:03:24.937519 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:24.937486    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36b03010-ss8rh"]
Oct 30 05:03:24.942649 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:24.942624    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:03:24.942795 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:24.942707    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\""
Oct 30 05:03:24.983355 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:24.983322    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msdnj\" (UniqueName: \"kubernetes.io/projected/504ddefe-3772-418a-8e89-39c1090cf6db-kube-api-access-msdnj\") pod \"maas-api-revoke-sub-36b03010-ss8rh\" (UID: \"504ddefe-3772-418a-8e89-39c1090cf6db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b03010-ss8rh"
Oct 30 05:03:24.983559 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:24.983460    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/504ddefe-3772-418a-8e89-39c1090cf6db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36b03010-ss8rh\" (UID: \"504ddefe-3772-418a-8e89-39c1090cf6db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b03010-ss8rh"
Oct 30 05:03:24.983559 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:24.983533    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/504ddefe-3772-418a-8e89-39c1090cf6db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36b03010-ss8rh\" (UID: \"504ddefe-3772-418a-8e89-39c1090cf6db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b03010-ss8rh"
Oct 30 05:03:25.084637 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:25.084603    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msdnj\" (UniqueName: \"kubernetes.io/projected/504ddefe-3772-418a-8e89-39c1090cf6db-kube-api-access-msdnj\") pod \"maas-api-revoke-sub-36b03010-ss8rh\" (UID: \"504ddefe-3772-418a-8e89-39c1090cf6db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b03010-ss8rh"
Oct 30 05:03:25.084812 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:25.084731    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/504ddefe-3772-418a-8e89-39c1090cf6db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36b03010-ss8rh\" (UID: \"504ddefe-3772-418a-8e89-39c1090cf6db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b03010-ss8rh"
Oct 30 05:03:25.084876 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:25.084826    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/504ddefe-3772-418a-8e89-39c1090cf6db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36b03010-ss8rh\" (UID: \"504ddefe-3772-418a-8e89-39c1090cf6db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b03010-ss8rh"
Oct 30 05:03:25.085436 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:25.085410    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/504ddefe-3772-418a-8e89-39c1090cf6db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36b03010-ss8rh\" (UID: \"504ddefe-3772-418a-8e89-39c1090cf6db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b03010-ss8rh"
Oct 30 05:03:25.087190 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:25.087172    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/504ddefe-3772-418a-8e89-39c1090cf6db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36b03010-ss8rh\" (UID: \"504ddefe-3772-418a-8e89-39c1090cf6db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b03010-ss8rh"
Oct 30 05:03:25.103795 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:25.103765    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msdnj\" (UniqueName: \"kubernetes.io/projected/504ddefe-3772-418a-8e89-39c1090cf6db-kube-api-access-msdnj\") pod \"maas-api-revoke-sub-36b03010-ss8rh\" (UID: \"504ddefe-3772-418a-8e89-39c1090cf6db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b03010-ss8rh"
Oct 30 05:03:25.198133 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:25.198059    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:25.234670 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:25.234634    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b03010-ss8rh"
Oct 30 05:03:25.555082 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:03:25.555045    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod504ddefe_3772_418a_8e89_39c1090cf6db.slice/crio-39d037c14df2ea38d5acf8a49c6dee5c70f0e80235f90e246e188889cc0f9e15 WatchSource:0}: Error finding container 39d037c14df2ea38d5acf8a49c6dee5c70f0e80235f90e246e188889cc0f9e15: Status 404 returned error can't find the container with id 39d037c14df2ea38d5acf8a49c6dee5c70f0e80235f90e246e188889cc0f9e15
Oct 30 05:03:25.555233 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:25.555125    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36b03010-ss8rh"]
Oct 30 05:03:25.590105 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:25.590081    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:25.684516 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:25.684480    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:25.781022 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:25.780984    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:25.934223 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:25.934188    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b03010-ss8rh" event={"ID":"504ddefe-3772-418a-8e89-39c1090cf6db","Type":"ContainerStarted","Data":"831f5ac3be0f661f540804bd487a42e7b3ef9093f1bc579dbf543ec7dd919c7f"}
Oct 30 05:03:25.934618 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:25.934237    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b03010-ss8rh" event={"ID":"504ddefe-3772-418a-8e89-39c1090cf6db","Type":"ContainerStarted","Data":"39d037c14df2ea38d5acf8a49c6dee5c70f0e80235f90e246e188889cc0f9e15"}
Oct 30 05:03:25.986695 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:25.986627    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b03010-ss8rh" podStartSLOduration=1.986607046 podStartE2EDuration="1.986607046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:03:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:03:25.981558361 +0000 UTC m=+1135.205378289" watchObservedRunningTime="2026-10-30 05:03:25.986607046 +0000 UTC m=+1135.210426948"
Oct 30 05:03:26.801892 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:26.801798    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:26.821801 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:03:26.821763    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:03:26.822386 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:26.822366    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d67de78b-67f5fbb5c5-dp59d"]
Oct 30 05:03:26.826602 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:26.826580    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d67de78b-67f5fbb5c5-dp59d"
Oct 30 05:03:26.846540 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:26.846512    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d67de78b-67f5fbb5c5-dp59d"]
Oct 30 05:03:26.851993 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:26.851970    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-d67de78b-serving-cert\""
Oct 30 05:03:26.893483 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:26.893419    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:26.911238 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:26.911211    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c79b9\" (UniqueName: \"kubernetes.io/projected/92cd0f2e-f7e5-47d3-84e7-aa96c8d75736-kube-api-access-c79b9\") pod \"maas-api-e2e-ait-d67de78b-67f5fbb5c5-dp59d\" (UID: \"92cd0f2e-f7e5-47d3-84e7-aa96c8d75736\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d67de78b-67f5fbb5c5-dp59d"
Oct 30 05:03:26.911447 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:26.911427    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92cd0f2e-f7e5-47d3-84e7-aa96c8d75736-metrics-certs\") pod \"maas-api-e2e-ait-d67de78b-67f5fbb5c5-dp59d\" (UID: \"92cd0f2e-f7e5-47d3-84e7-aa96c8d75736\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d67de78b-67f5fbb5c5-dp59d"
Oct 30 05:03:26.911537 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:26.911499    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92cd0f2e-f7e5-47d3-84e7-aa96c8d75736-maas-api-tls\") pod \"maas-api-e2e-ait-d67de78b-67f5fbb5c5-dp59d\" (UID: \"92cd0f2e-f7e5-47d3-84e7-aa96c8d75736\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d67de78b-67f5fbb5c5-dp59d"
Oct 30 05:03:26.911600 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:26.911579    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92cd0f2e-f7e5-47d3-84e7-aa96c8d75736-openshift-service-ca\") pod \"maas-api-e2e-ait-d67de78b-67f5fbb5c5-dp59d\" (UID: \"92cd0f2e-f7e5-47d3-84e7-aa96c8d75736\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d67de78b-67f5fbb5c5-dp59d"
Oct 30 05:03:26.944233 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:26.944205    5898 generic.go:356] "Generic (PLEG): container finished" podID="504ddefe-3772-418a-8e89-39c1090cf6db" containerID="831f5ac3be0f661f540804bd487a42e7b3ef9093f1bc579dbf543ec7dd919c7f" exitCode=0
Oct 30 05:03:26.944678 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:26.944282    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b03010-ss8rh" event={"ID":"504ddefe-3772-418a-8e89-39c1090cf6db","Type":"ContainerDied","Data":"831f5ac3be0f661f540804bd487a42e7b3ef9093f1bc579dbf543ec7dd919c7f"}
Oct 30 05:03:26.978983 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:26.978887    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:27.012731 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:27.012698    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92cd0f2e-f7e5-47d3-84e7-aa96c8d75736-metrics-certs\") pod \"maas-api-e2e-ait-d67de78b-67f5fbb5c5-dp59d\" (UID: \"92cd0f2e-f7e5-47d3-84e7-aa96c8d75736\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d67de78b-67f5fbb5c5-dp59d"
Oct 30 05:03:27.012914 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:27.012781    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92cd0f2e-f7e5-47d3-84e7-aa96c8d75736-maas-api-tls\") pod \"maas-api-e2e-ait-d67de78b-67f5fbb5c5-dp59d\" (UID: \"92cd0f2e-f7e5-47d3-84e7-aa96c8d75736\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d67de78b-67f5fbb5c5-dp59d"
Oct 30 05:03:27.012914 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:27.012820    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92cd0f2e-f7e5-47d3-84e7-aa96c8d75736-openshift-service-ca\") pod \"maas-api-e2e-ait-d67de78b-67f5fbb5c5-dp59d\" (UID: \"92cd0f2e-f7e5-47d3-84e7-aa96c8d75736\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d67de78b-67f5fbb5c5-dp59d"
Oct 30 05:03:27.012914 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:27.012880    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c79b9\" (UniqueName: \"kubernetes.io/projected/92cd0f2e-f7e5-47d3-84e7-aa96c8d75736-kube-api-access-c79b9\") pod \"maas-api-e2e-ait-d67de78b-67f5fbb5c5-dp59d\" (UID: \"92cd0f2e-f7e5-47d3-84e7-aa96c8d75736\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d67de78b-67f5fbb5c5-dp59d"
Oct 30 05:03:27.013635 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:27.013608    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92cd0f2e-f7e5-47d3-84e7-aa96c8d75736-openshift-service-ca\") pod \"maas-api-e2e-ait-d67de78b-67f5fbb5c5-dp59d\" (UID: \"92cd0f2e-f7e5-47d3-84e7-aa96c8d75736\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d67de78b-67f5fbb5c5-dp59d"
Oct 30 05:03:27.015421 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:27.015395    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92cd0f2e-f7e5-47d3-84e7-aa96c8d75736-maas-api-tls\") pod \"maas-api-e2e-ait-d67de78b-67f5fbb5c5-dp59d\" (UID: \"92cd0f2e-f7e5-47d3-84e7-aa96c8d75736\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d67de78b-67f5fbb5c5-dp59d"
Oct 30 05:03:27.015517 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:27.015395    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92cd0f2e-f7e5-47d3-84e7-aa96c8d75736-metrics-certs\") pod \"maas-api-e2e-ait-d67de78b-67f5fbb5c5-dp59d\" (UID: \"92cd0f2e-f7e5-47d3-84e7-aa96c8d75736\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d67de78b-67f5fbb5c5-dp59d"
Oct 30 05:03:27.032684 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:27.032657    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c79b9\" (UniqueName: \"kubernetes.io/projected/92cd0f2e-f7e5-47d3-84e7-aa96c8d75736-kube-api-access-c79b9\") pod \"maas-api-e2e-ait-d67de78b-67f5fbb5c5-dp59d\" (UID: \"92cd0f2e-f7e5-47d3-84e7-aa96c8d75736\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d67de78b-67f5fbb5c5-dp59d"
Oct 30 05:03:27.078761 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:27.078732    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:27.141271 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:27.141231    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d67de78b-67f5fbb5c5-dp59d"
Oct 30 05:03:27.544489 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:03:27.544462    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92cd0f2e_f7e5_47d3_84e7_aa96c8d75736.slice/crio-7f2cbffd93c21fb3fa83d45a6bd79c1769f16140ea50ec3c47fa65f388f63633 WatchSource:0}: Error finding container 7f2cbffd93c21fb3fa83d45a6bd79c1769f16140ea50ec3c47fa65f388f63633: Status 404 returned error can't find the container with id 7f2cbffd93c21fb3fa83d45a6bd79c1769f16140ea50ec3c47fa65f388f63633
Oct 30 05:03:27.545875 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:27.545845    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d67de78b-67f5fbb5c5-dp59d"]
Oct 30 05:03:27.959921 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:27.959373    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d67de78b-67f5fbb5c5-dp59d" event={"ID":"92cd0f2e-f7e5-47d3-84e7-aa96c8d75736","Type":"ContainerStarted","Data":"7f2cbffd93c21fb3fa83d45a6bd79c1769f16140ea50ec3c47fa65f388f63633"}
Oct 30 05:03:28.149052 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:28.149022    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b03010-ss8rh"
Oct 30 05:03:28.231119 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:28.231038    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/504ddefe-3772-418a-8e89-39c1090cf6db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/504ddefe-3772-418a-8e89-39c1090cf6db-maas-api-openshift-ca\") pod \"504ddefe-3772-418a-8e89-39c1090cf6db\" (UID: \"504ddefe-3772-418a-8e89-39c1090cf6db\") "
Oct 30 05:03:28.231286 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:28.231169    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/504ddefe-3772-418a-8e89-39c1090cf6db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/504ddefe-3772-418a-8e89-39c1090cf6db-maas-api-xks-ca\") pod \"504ddefe-3772-418a-8e89-39c1090cf6db\" (UID: \"504ddefe-3772-418a-8e89-39c1090cf6db\") "
Oct 30 05:03:28.231286 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:28.231256    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/504ddefe-3772-418a-8e89-39c1090cf6db-kube-api-access-msdnj\" (UniqueName: \"kubernetes.io/projected/504ddefe-3772-418a-8e89-39c1090cf6db-kube-api-access-msdnj\") pod \"504ddefe-3772-418a-8e89-39c1090cf6db\" (UID: \"504ddefe-3772-418a-8e89-39c1090cf6db\") "
Oct 30 05:03:28.231472 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:28.231448    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/504ddefe-3772-418a-8e89-39c1090cf6db-maas-api-openshift-ca" pod "504ddefe-3772-418a-8e89-39c1090cf6db" (UID: "504ddefe-3772-418a-8e89-39c1090cf6db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:03:28.231754 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:28.231730    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/504ddefe-3772-418a-8e89-39c1090cf6db-maas-api-openshift-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:03:28.233655 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:28.233627    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/504ddefe-3772-418a-8e89-39c1090cf6db-kube-api-access-msdnj" pod "504ddefe-3772-418a-8e89-39c1090cf6db" (UID: "504ddefe-3772-418a-8e89-39c1090cf6db"). InnerVolumeSpecName "kube-api-access-msdnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:03:28.242243 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:28.242215    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/504ddefe-3772-418a-8e89-39c1090cf6db-maas-api-xks-ca" pod "504ddefe-3772-418a-8e89-39c1090cf6db" (UID: "504ddefe-3772-418a-8e89-39c1090cf6db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:03:28.297689 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:28.297658    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-b0fba292-65679f777d-tkn74"
Oct 30 05:03:28.333704 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:28.333674    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msdnj\" (UniqueName: \"kubernetes.io/projected/504ddefe-3772-418a-8e89-39c1090cf6db-kube-api-access-msdnj\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:03:28.333704 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:28.333699    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/504ddefe-3772-418a-8e89-39c1090cf6db-maas-api-xks-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:03:28.884254 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:28.884220    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:28.969515 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:28.969488    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b03010-ss8rh"
Oct 30 05:03:28.970024 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:28.969520    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b03010-ss8rh" event={"ID":"504ddefe-3772-418a-8e89-39c1090cf6db","Type":"ContainerDied","Data":"39d037c14df2ea38d5acf8a49c6dee5c70f0e80235f90e246e188889cc0f9e15"}
Oct 30 05:03:28.970024 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:28.969569    5898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39d037c14df2ea38d5acf8a49c6dee5c70f0e80235f90e246e188889cc0f9e15"
Oct 30 05:03:28.971604 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:28.971569    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d67de78b-67f5fbb5c5-dp59d" event={"ID":"92cd0f2e-f7e5-47d3-84e7-aa96c8d75736","Type":"ContainerStarted","Data":"931194ac5f53a2d20dec2b97e762479428867976615a0b65efb38fbb427e5a4d"}
Oct 30 05:03:28.971774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:28.971752    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d67de78b-67f5fbb5c5-dp59d"
Oct 30 05:03:28.971893 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:28.971876    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:29.015613 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:29.015565    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d67de78b-67f5fbb5c5-dp59d" podStartSLOduration=2.553350568 podStartE2EDuration="3.015552197s" totalImagesPullingTime="462.201629ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:03:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:03:29.012439337 +0000 UTC m=+1138.236259259" watchObservedRunningTime="2026-10-30 05:03:29.015552197 +0000 UTC m=+1138.239372127"
Oct 30 05:03:30.759493 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:30.759441    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:32.360828 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:32.360790    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:33.984096 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:33.984059    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:34.663590 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:34.663558    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:34.983415 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:34.983346    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d67de78b-67f5fbb5c5-dp59d"
Oct 30 05:03:35.037886 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:35.037846    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00fda0ea-b5tnk"]
Oct 30 05:03:35.042698 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:35.042665    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00fda0ea-b5tnk"]
Oct 30 05:03:35.589080 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:35.589048    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:03:35.662699 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:35.662660    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b0fba292-c7b9d9646-l4fz5"]
Oct 30 05:03:35.663384 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:35.663361    5898 cpu_manager.go:414] "containerMap: removing container" podUID="504ddefe-3772-418a-8e89-39c1090cf6db" containerName="revoke-keys"
Oct 30 05:03:35.663384 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:35.663381    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="504ddefe-3772-418a-8e89-39c1090cf6db" containerName="revoke-keys"
Oct 30 05:03:35.663531 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:35.663501    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="504ddefe-3772-418a-8e89-39c1090cf6db" containerName="revoke-keys"
Oct 30 05:03:35.672728 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:35.672698    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0fba292-c7b9d9646-l4fz5"
Oct 30 05:03:35.684731 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:35.684706    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b0fba292-c7b9d9646-l4fz5"]
Oct 30 05:03:35.688230 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:35.688205    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-b0fba292-serving-cert\""
Oct 30 05:03:35.733127 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:35.733088    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2163caad-3db0-4f6f-9e99-f3c0a83cb50e-metrics-certs\") pod \"maas-api-e2e-ait-b0fba292-c7b9d9646-l4fz5\" (UID: \"2163caad-3db0-4f6f-9e99-f3c0a83cb50e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0fba292-c7b9d9646-l4fz5"
Oct 30 05:03:35.733312 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:35.733152    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2163caad-3db0-4f6f-9e99-f3c0a83cb50e-maas-api-tls\") pod \"maas-api-e2e-ait-b0fba292-c7b9d9646-l4fz5\" (UID: \"2163caad-3db0-4f6f-9e99-f3c0a83cb50e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0fba292-c7b9d9646-l4fz5"
Oct 30 05:03:35.733312 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:35.733286    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2qkr\" (UniqueName: \"kubernetes.io/projected/2163caad-3db0-4f6f-9e99-f3c0a83cb50e-kube-api-access-f2qkr\") pod \"maas-api-e2e-ait-b0fba292-c7b9d9646-l4fz5\" (UID: \"2163caad-3db0-4f6f-9e99-f3c0a83cb50e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0fba292-c7b9d9646-l4fz5"
Oct 30 05:03:35.733441 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:35.733356    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2163caad-3db0-4f6f-9e99-f3c0a83cb50e-openshift-service-ca\") pod \"maas-api-e2e-ait-b0fba292-c7b9d9646-l4fz5\" (UID: \"2163caad-3db0-4f6f-9e99-f3c0a83cb50e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0fba292-c7b9d9646-l4fz5"
Oct 30 05:03:35.744285 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:35.744252    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:35.834747 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:35.834706    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2163caad-3db0-4f6f-9e99-f3c0a83cb50e-metrics-certs\") pod \"maas-api-e2e-ait-b0fba292-c7b9d9646-l4fz5\" (UID: \"2163caad-3db0-4f6f-9e99-f3c0a83cb50e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0fba292-c7b9d9646-l4fz5"
Oct 30 05:03:35.834956 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:35.834770    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2163caad-3db0-4f6f-9e99-f3c0a83cb50e-maas-api-tls\") pod \"maas-api-e2e-ait-b0fba292-c7b9d9646-l4fz5\" (UID: \"2163caad-3db0-4f6f-9e99-f3c0a83cb50e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0fba292-c7b9d9646-l4fz5"
Oct 30 05:03:35.834956 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:35.834831    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2qkr\" (UniqueName: \"kubernetes.io/projected/2163caad-3db0-4f6f-9e99-f3c0a83cb50e-kube-api-access-f2qkr\") pod \"maas-api-e2e-ait-b0fba292-c7b9d9646-l4fz5\" (UID: \"2163caad-3db0-4f6f-9e99-f3c0a83cb50e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0fba292-c7b9d9646-l4fz5"
Oct 30 05:03:35.834956 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:35.834895    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2163caad-3db0-4f6f-9e99-f3c0a83cb50e-openshift-service-ca\") pod \"maas-api-e2e-ait-b0fba292-c7b9d9646-l4fz5\" (UID: \"2163caad-3db0-4f6f-9e99-f3c0a83cb50e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0fba292-c7b9d9646-l4fz5"
Oct 30 05:03:35.835527 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:35.835502    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2163caad-3db0-4f6f-9e99-f3c0a83cb50e-openshift-service-ca\") pod \"maas-api-e2e-ait-b0fba292-c7b9d9646-l4fz5\" (UID: \"2163caad-3db0-4f6f-9e99-f3c0a83cb50e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0fba292-c7b9d9646-l4fz5"
Oct 30 05:03:35.837259 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:35.837237    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2163caad-3db0-4f6f-9e99-f3c0a83cb50e-metrics-certs\") pod \"maas-api-e2e-ait-b0fba292-c7b9d9646-l4fz5\" (UID: \"2163caad-3db0-4f6f-9e99-f3c0a83cb50e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0fba292-c7b9d9646-l4fz5"
Oct 30 05:03:35.837359 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:35.837245    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2163caad-3db0-4f6f-9e99-f3c0a83cb50e-maas-api-tls\") pod \"maas-api-e2e-ait-b0fba292-c7b9d9646-l4fz5\" (UID: \"2163caad-3db0-4f6f-9e99-f3c0a83cb50e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0fba292-c7b9d9646-l4fz5"
Oct 30 05:03:35.845121 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:35.845054    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:35.853226 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:35.853199    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2qkr\" (UniqueName: \"kubernetes.io/projected/2163caad-3db0-4f6f-9e99-f3c0a83cb50e-kube-api-access-f2qkr\") pod \"maas-api-e2e-ait-b0fba292-c7b9d9646-l4fz5\" (UID: \"2163caad-3db0-4f6f-9e99-f3c0a83cb50e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0fba292-c7b9d9646-l4fz5"
Oct 30 05:03:35.940695 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:35.940652    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:35.989267 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:35.989232    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0fba292-c7b9d9646-l4fz5"
Oct 30 05:03:36.299763 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:03:36.299732    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2163caad_3db0_4f6f_9e99_f3c0a83cb50e.slice/crio-25bcd98972330a384f4dd62e5e1df264005dd12285963cf8755e110fea55d894 WatchSource:0}: Error finding container 25bcd98972330a384f4dd62e5e1df264005dd12285963cf8755e110fea55d894: Status 404 returned error can't find the container with id 25bcd98972330a384f4dd62e5e1df264005dd12285963cf8755e110fea55d894
Oct 30 05:03:36.300278 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:36.300254    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b0fba292-c7b9d9646-l4fz5"]
Oct 30 05:03:36.920818 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:36.920701    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01f80893-ab5c-41c6-8f8d-4e578bfb0feb" path="/var/lib/kubelet/pods/01f80893-ab5c-41c6-8f8d-4e578bfb0feb/volumes"
Oct 30 05:03:37.086392 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:37.086354    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0fba292-c7b9d9646-l4fz5" event={"ID":"2163caad-3db0-4f6f-9e99-f3c0a83cb50e","Type":"ContainerStarted","Data":"56efb6814892c5e8b5c628cc197adbe38fa531fa26529f6d809fb5ddde8108b9"}
Oct 30 05:03:37.086662 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:37.086643    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0fba292-c7b9d9646-l4fz5" event={"ID":"2163caad-3db0-4f6f-9e99-f3c0a83cb50e","Type":"ContainerStarted","Data":"25bcd98972330a384f4dd62e5e1df264005dd12285963cf8755e110fea55d894"}
Oct 30 05:03:37.087741 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:37.087719    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0fba292-c7b9d9646-l4fz5"
Oct 30 05:03:37.160928 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:37.152414    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0fba292-c7b9d9646-l4fz5" podStartSLOduration=1.838542438 podStartE2EDuration="2.152394536s" totalImagesPullingTime="313.852098ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:03:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:03:37.152120147 +0000 UTC m=+1146.375940073" watchObservedRunningTime="2026-10-30 05:03:37.152394536 +0000 UTC m=+1146.376214438"
Oct 30 05:03:37.961923 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:37.956156    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6" podUID="47ff272a-ac5c-4f59-bf7b-5c0692d09386" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:03:37.965383 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:37.964534    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9" podUID="1647f909-5580-4219-a2e1-30b4cd994379" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 05:03:38.003208 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:38.003100    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49" podUID="05eef4eb-f379-4dc7-a947-123bd85a4968" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:03:38.046417 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:38.046011    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb" podUID="6f16bd6a-a7f0-49bb-844f-57b777bde2ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:03:38.355949 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:38.354995    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g" podUID="70294a85-1050-4632-b3bf-c3e44a4c0b41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:03:38.365035 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:38.365000    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6" podUID="b09804e3-8135-4f6d-ae34-c45d6da9728e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:03:38.962319 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:38.960001    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6" podUID="47ff272a-ac5c-4f59-bf7b-5c0692d09386" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:03:38.962319 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:38.960622    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6"
Oct 30 05:03:39.018889 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:39.017018    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9" podUID="1647f909-5580-4219-a2e1-30b4cd994379" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:03:39.018889 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:39.017484    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9"
Oct 30 05:03:39.080927 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:39.075108    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx" podUID="d523008f-c740-4a41-89cb-e82e1ffa0094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:03:39.236985 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:39.118858    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb" podUID="6f16bd6a-a7f0-49bb-844f-57b777bde2ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:03:39.236985 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:39.119425    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb"
Oct 30 05:03:39.425926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:39.393183    5898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9" podUID="1647f909-5580-4219-a2e1-30b4cd994379" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": context deadline exceeded"
Oct 30 05:03:39.425926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:39.393293    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g" podUID="70294a85-1050-4632-b3bf-c3e44a4c0b41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 05:03:39.425926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:39.394103    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g"
Oct 30 05:03:39.425926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:39.415118    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6" podUID="b09804e3-8135-4f6d-ae34-c45d6da9728e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 05:03:39.425926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:39.415586    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6"
Oct 30 05:03:39.428914 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:39.428862    5898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5" podUID="ebf8f4a2-8ac6-48ec-bdba-3f6a99653264" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:03:39.465277 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:39.430569    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh" podUID="38428a53-04c7-400a-9c78-52407e982478" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:03:39.478630 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:39.478598    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:39.772926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:39.772372    5898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849" podUID="a9707f50-d5e1-409f-b8f3-c58629677aaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": context deadline exceeded"
Oct 30 05:03:39.900407 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:39.900358    5898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g" podUID="70294a85-1050-4632-b3bf-c3e44a4c0b41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:03:39.976966 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:39.970665    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58" podUID="c27f3b21-1447-4af9-8d99-c04fa9b23cda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:03:39.976966 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:39.972824    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849" podUID="a9707f50-d5e1-409f-b8f3-c58629677aaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 05:03:40.010849 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:39.994150    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6" podUID="47ff272a-ac5c-4f59-bf7b-5c0692d09386" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:03:40.011996 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:40.011931    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49" podUID="05eef4eb-f379-4dc7-a947-123bd85a4968" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 05:03:40.029305 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:40.024790    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9" podUID="1647f909-5580-4219-a2e1-30b4cd994379" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 05:03:40.084965 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:40.084924    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx" podUID="d523008f-c740-4a41-89cb-e82e1ffa0094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 05:03:40.085681 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:40.085659    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx"
Oct 30 05:03:40.107781 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:40.107733    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5" podUID="ebf8f4a2-8ac6-48ec-bdba-3f6a99653264" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 05:03:40.137940 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:40.120312    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb" podUID="6f16bd6a-a7f0-49bb-844f-57b777bde2ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:03:40.397786 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:40.397758    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:40.429726 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:40.429654    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g" podUID="70294a85-1050-4632-b3bf-c3e44a4c0b41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:03:40.436944 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:40.436887    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247" podUID="1d865d3e-a6e9-407e-844e-cc75fc7f1354" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:03:40.438827 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:40.438799    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6" podUID="b09804e3-8135-4f6d-ae34-c45d6da9728e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 05:03:40.465678 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:40.465637    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh" podUID="38428a53-04c7-400a-9c78-52407e982478" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:03:40.467204 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:40.467181    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh"
Oct 30 05:03:40.532547 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:40.532470    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:41.081807 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:41.081298    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49" podUID="05eef4eb-f379-4dc7-a947-123bd85a4968" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:03:41.081807 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:41.081660    5898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6" podUID="b09804e3-8135-4f6d-ae34-c45d6da9728e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:03:41.105741 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:41.085190    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58" podUID="c27f3b21-1447-4af9-8d99-c04fa9b23cda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 05:03:41.117926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:41.105991    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx" podUID="d523008f-c740-4a41-89cb-e82e1ffa0094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 05:03:41.117926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:41.108272    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5" podUID="ebf8f4a2-8ac6-48ec-bdba-3f6a99653264" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 05:03:41.127926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:41.127059    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb" podUID="6f16bd6a-a7f0-49bb-844f-57b777bde2ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:03:41.128568 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:41.128164    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6" podUID="47ff272a-ac5c-4f59-bf7b-5c0692d09386" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:03:41.288835 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:41.288803    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9"
Oct 30 05:03:41.289172 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:41.289154    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49"
Oct 30 05:03:41.289373 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:41.289350    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58"
Oct 30 05:03:41.289531 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:41.289518    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5"
Oct 30 05:03:41.418603 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:41.418507    5898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6" podUID="47ff272a-ac5c-4f59-bf7b-5c0692d09386" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": context deadline exceeded"
Oct 30 05:03:41.446854 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:41.446816    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g" podUID="70294a85-1050-4632-b3bf-c3e44a4c0b41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 05:03:41.484995 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:41.450994    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6" podUID="b09804e3-8135-4f6d-ae34-c45d6da9728e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:03:41.484995 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:41.451023    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247" podUID="1d865d3e-a6e9-407e-844e-cc75fc7f1354" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 05:03:41.484995 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:41.474311    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247"
Oct 30 05:03:41.519040 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:41.518031    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh" podUID="38428a53-04c7-400a-9c78-52407e982478" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:03:41.918191 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:41.918149    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb"
Oct 30 05:03:42.036105 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:42.036058    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849" podUID="a9707f50-d5e1-409f-b8f3-c58629677aaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 05:03:42.036920 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:42.036004    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9" podUID="1647f909-5580-4219-a2e1-30b4cd994379" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:03:42.109923 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:42.102013    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49" podUID="05eef4eb-f379-4dc7-a947-123bd85a4968" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 05:03:42.109923 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:42.103233    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58" podUID="c27f3b21-1447-4af9-8d99-c04fa9b23cda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:03:42.136947 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:42.117306    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx" podUID="d523008f-c740-4a41-89cb-e82e1ffa0094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 05:03:42.136947 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:42.117619    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5" podUID="ebf8f4a2-8ac6-48ec-bdba-3f6a99653264" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 05:03:42.136947 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:42.129879    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6" podUID="47ff272a-ac5c-4f59-bf7b-5c0692d09386" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:03:42.136947 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:42.136789    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:42.437758 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:42.437724    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:42.475413 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:42.475367    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247" podUID="1d865d3e-a6e9-407e-844e-cc75fc7f1354" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 05:03:42.479923 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:42.476143    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6" podUID="b09804e3-8135-4f6d-ae34-c45d6da9728e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 05:03:42.490760 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:42.480333    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g" podUID="70294a85-1050-4632-b3bf-c3e44a4c0b41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:03:42.519402 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:42.518332    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh" podUID="38428a53-04c7-400a-9c78-52407e982478" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:03:42.712088 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:42.709008    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-0b21ba/distinct-w5-0b21ba-kserve-7674c59666-m2sbd"]
Oct 30 05:03:42.716796 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:42.716771    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-0b21ba/distinct-w5-0b21ba-kserve-7674c59666-m2sbd"
Oct 30 05:03:42.717489 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:42.716893    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-0b21ba/distinct-w5-0b21ba-kserve-7674c59666-m2sbd"]
Oct 30 05:03:42.758433 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:42.757108    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-0b21ba\"/\"distinct-w5-0b21ba-kserve-self-signed-certs\""
Oct 30 05:03:42.797388 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:42.797348    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24c6b2d4-3377-4079-8273-3f48064c3a98-tls-certs\") pod \"distinct-w5-0b21ba-kserve-7674c59666-m2sbd\" (UID: \"24c6b2d4-3377-4079-8273-3f48064c3a98\") " pod="e2e-models-e2e-worker-w5-0b21ba/distinct-w5-0b21ba-kserve-7674c59666-m2sbd"
Oct 30 05:03:42.797756 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:42.797730    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24c6b2d4-3377-4079-8273-3f48064c3a98-tmp-dir\") pod \"distinct-w5-0b21ba-kserve-7674c59666-m2sbd\" (UID: \"24c6b2d4-3377-4079-8273-3f48064c3a98\") " pod="e2e-models-e2e-worker-w5-0b21ba/distinct-w5-0b21ba-kserve-7674c59666-m2sbd"
Oct 30 05:03:42.798038 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:42.798019    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24c6b2d4-3377-4079-8273-3f48064c3a98-model-cache\") pod \"distinct-w5-0b21ba-kserve-7674c59666-m2sbd\" (UID: \"24c6b2d4-3377-4079-8273-3f48064c3a98\") " pod="e2e-models-e2e-worker-w5-0b21ba/distinct-w5-0b21ba-kserve-7674c59666-m2sbd"
Oct 30 05:03:42.798261 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:42.798241    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24c6b2d4-3377-4079-8273-3f48064c3a98-home\") pod \"distinct-w5-0b21ba-kserve-7674c59666-m2sbd\" (UID: \"24c6b2d4-3377-4079-8273-3f48064c3a98\") " pod="e2e-models-e2e-worker-w5-0b21ba/distinct-w5-0b21ba-kserve-7674c59666-m2sbd"
Oct 30 05:03:42.798459 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:42.798441    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cblr\" (UniqueName: \"kubernetes.io/projected/24c6b2d4-3377-4079-8273-3f48064c3a98-kube-api-access-8cblr\") pod \"distinct-w5-0b21ba-kserve-7674c59666-m2sbd\" (UID: \"24c6b2d4-3377-4079-8273-3f48064c3a98\") " pod="e2e-models-e2e-worker-w5-0b21ba/distinct-w5-0b21ba-kserve-7674c59666-m2sbd"
Oct 30 05:03:42.802274 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:42.802251    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24c6b2d4-3377-4079-8273-3f48064c3a98-dshm\") pod \"distinct-w5-0b21ba-kserve-7674c59666-m2sbd\" (UID: \"24c6b2d4-3377-4079-8273-3f48064c3a98\") " pod="e2e-models-e2e-worker-w5-0b21ba/distinct-w5-0b21ba-kserve-7674c59666-m2sbd"
Oct 30 05:03:42.834577 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:42.834545    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49"
Oct 30 05:03:42.905936 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:42.905238    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24c6b2d4-3377-4079-8273-3f48064c3a98-home\") pod \"distinct-w5-0b21ba-kserve-7674c59666-m2sbd\" (UID: \"24c6b2d4-3377-4079-8273-3f48064c3a98\") " pod="e2e-models-e2e-worker-w5-0b21ba/distinct-w5-0b21ba-kserve-7674c59666-m2sbd"
Oct 30 05:03:42.905936 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:42.905334    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cblr\" (UniqueName: \"kubernetes.io/projected/24c6b2d4-3377-4079-8273-3f48064c3a98-kube-api-access-8cblr\") pod \"distinct-w5-0b21ba-kserve-7674c59666-m2sbd\" (UID: \"24c6b2d4-3377-4079-8273-3f48064c3a98\") " pod="e2e-models-e2e-worker-w5-0b21ba/distinct-w5-0b21ba-kserve-7674c59666-m2sbd"
Oct 30 05:03:42.905936 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:42.905399    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24c6b2d4-3377-4079-8273-3f48064c3a98-dshm\") pod \"distinct-w5-0b21ba-kserve-7674c59666-m2sbd\" (UID: \"24c6b2d4-3377-4079-8273-3f48064c3a98\") " pod="e2e-models-e2e-worker-w5-0b21ba/distinct-w5-0b21ba-kserve-7674c59666-m2sbd"
Oct 30 05:03:42.905936 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:42.905701    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24c6b2d4-3377-4079-8273-3f48064c3a98-tls-certs\") pod \"distinct-w5-0b21ba-kserve-7674c59666-m2sbd\" (UID: \"24c6b2d4-3377-4079-8273-3f48064c3a98\") " pod="e2e-models-e2e-worker-w5-0b21ba/distinct-w5-0b21ba-kserve-7674c59666-m2sbd"
Oct 30 05:03:42.905936 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:42.905781    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24c6b2d4-3377-4079-8273-3f48064c3a98-tmp-dir\") pod \"distinct-w5-0b21ba-kserve-7674c59666-m2sbd\" (UID: \"24c6b2d4-3377-4079-8273-3f48064c3a98\") " pod="e2e-models-e2e-worker-w5-0b21ba/distinct-w5-0b21ba-kserve-7674c59666-m2sbd"
Oct 30 05:03:42.905936 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:42.905872    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24c6b2d4-3377-4079-8273-3f48064c3a98-model-cache\") pod \"distinct-w5-0b21ba-kserve-7674c59666-m2sbd\" (UID: \"24c6b2d4-3377-4079-8273-3f48064c3a98\") " pod="e2e-models-e2e-worker-w5-0b21ba/distinct-w5-0b21ba-kserve-7674c59666-m2sbd"
Oct 30 05:03:42.912919 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:42.906639    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24c6b2d4-3377-4079-8273-3f48064c3a98-home\") pod \"distinct-w5-0b21ba-kserve-7674c59666-m2sbd\" (UID: \"24c6b2d4-3377-4079-8273-3f48064c3a98\") " pod="e2e-models-e2e-worker-w5-0b21ba/distinct-w5-0b21ba-kserve-7674c59666-m2sbd"
Oct 30 05:03:42.912919 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:42.907638    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24c6b2d4-3377-4079-8273-3f48064c3a98-model-cache\") pod \"distinct-w5-0b21ba-kserve-7674c59666-m2sbd\" (UID: \"24c6b2d4-3377-4079-8273-3f48064c3a98\") " pod="e2e-models-e2e-worker-w5-0b21ba/distinct-w5-0b21ba-kserve-7674c59666-m2sbd"
Oct 30 05:03:42.912919 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:42.908179    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24c6b2d4-3377-4079-8273-3f48064c3a98-tmp-dir\") pod \"distinct-w5-0b21ba-kserve-7674c59666-m2sbd\" (UID: \"24c6b2d4-3377-4079-8273-3f48064c3a98\") " pod="e2e-models-e2e-worker-w5-0b21ba/distinct-w5-0b21ba-kserve-7674c59666-m2sbd"
Oct 30 05:03:43.121926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:43.121853    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5" podUID="ebf8f4a2-8ac6-48ec-bdba-3f6a99653264" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:03:43.159942 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:43.137131    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx" podUID="d523008f-c740-4a41-89cb-e82e1ffa0094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 05:03:43.159942 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:43.149727    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58" podUID="c27f3b21-1447-4af9-8d99-c04fa9b23cda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:03:43.176492 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:43.176444    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb" podUID="6f16bd6a-a7f0-49bb-844f-57b777bde2ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:03:43.188275 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:43.188238    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9" podUID="1647f909-5580-4219-a2e1-30b4cd994379" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 05:03:43.189336 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:43.189307    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6" podUID="47ff272a-ac5c-4f59-bf7b-5c0692d09386" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Oct 30 05:03:43.191257 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:43.191235    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cblr\" (UniqueName: \"kubernetes.io/projected/24c6b2d4-3377-4079-8273-3f48064c3a98-kube-api-access-8cblr\") pod \"distinct-w5-0b21ba-kserve-7674c59666-m2sbd\" (UID: \"24c6b2d4-3377-4079-8273-3f48064c3a98\") " pod="e2e-models-e2e-worker-w5-0b21ba/distinct-w5-0b21ba-kserve-7674c59666-m2sbd"
Oct 30 05:03:43.197734 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:43.197710    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24c6b2d4-3377-4079-8273-3f48064c3a98-tls-certs\") pod \"distinct-w5-0b21ba-kserve-7674c59666-m2sbd\" (UID: \"24c6b2d4-3377-4079-8273-3f48064c3a98\") " pod="e2e-models-e2e-worker-w5-0b21ba/distinct-w5-0b21ba-kserve-7674c59666-m2sbd"
Oct 30 05:03:43.362176 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:43.362140    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24c6b2d4-3377-4079-8273-3f48064c3a98-dshm\") pod \"distinct-w5-0b21ba-kserve-7674c59666-m2sbd\" (UID: \"24c6b2d4-3377-4079-8273-3f48064c3a98\") " pod="e2e-models-e2e-worker-w5-0b21ba/distinct-w5-0b21ba-kserve-7674c59666-m2sbd"
Oct 30 05:03:43.363044 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:43.363019    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247"
Oct 30 05:03:43.363705 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:43.363687    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9"
Oct 30 05:03:43.371450 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:43.371209    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-0b21ba/distinct-w5-0b21ba-kserve-7674c59666-m2sbd"
Oct 30 05:03:43.418652 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:43.418353    5898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx" podUID="d523008f-c740-4a41-89cb-e82e1ffa0094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:03:43.418882 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:43.418850    5898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh" podUID="38428a53-04c7-400a-9c78-52407e982478" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:03:43.474593 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:43.466806    5898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb" podUID="6f16bd6a-a7f0-49bb-844f-57b777bde2ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": context deadline exceeded"
Oct 30 05:03:43.498317 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:43.495282    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g" podUID="70294a85-1050-4632-b3bf-c3e44a4c0b41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:03:43.498579 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:43.498528    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6" podUID="b09804e3-8135-4f6d-ae34-c45d6da9728e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:03:43.526619 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:43.522121    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh" podUID="38428a53-04c7-400a-9c78-52407e982478" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 30 05:03:44.055933 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:44.052595    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6"
Oct 30 05:03:44.149859 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:44.149709    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx" podUID="d523008f-c740-4a41-89cb-e82e1ffa0094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 05:03:44.152355 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:44.151363    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5" podUID="ebf8f4a2-8ac6-48ec-bdba-3f6a99653264" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 05:03:44.154321 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:44.153956    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58" podUID="c27f3b21-1447-4af9-8d99-c04fa9b23cda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:03:44.362705 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:44.360938    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6"
Oct 30 05:03:44.365058 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:44.363013    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9" podUID="1647f909-5580-4219-a2e1-30b4cd994379" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 05:03:44.498791 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:44.498287    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g" podUID="70294a85-1050-4632-b3bf-c3e44a4c0b41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:03:44.516918 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:44.510630    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx"
Oct 30 05:03:44.522626 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:44.522590    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh" podUID="38428a53-04c7-400a-9c78-52407e982478" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:03:44.523370 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:44.523171    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-0b21ba/distinct-w5-0b21ba-kserve-7674c59666-m2sbd"]
Oct 30 05:03:44.784971 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:44.763701    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5"
Oct 30 05:03:44.784971 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:44.773340    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58"
Oct 30 05:03:44.787061 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:44.787036    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9"
Oct 30 05:03:45.183999 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:45.183645    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0fba292-c7b9d9646-l4fz5"
Oct 30 05:03:45.183999 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:45.183716    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g"
Oct 30 05:03:45.183999 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:45.183747    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh"
Oct 30 05:03:45.980912 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:45.980860    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-0b21ba/distinct-w5-0b21ba-kserve-7674c59666-m2sbd" event={"ID":"24c6b2d4-3377-4079-8273-3f48064c3a98","Type":"ContainerStarted","Data":"762151b43e59b658a751c9ded7fc81204ed50a26626d7edea28be74b3b45d23f"}
Oct 30 05:03:45.981082 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:45.980929    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-0b21ba/distinct-w5-0b21ba-kserve-7674c59666-m2sbd" event={"ID":"24c6b2d4-3377-4079-8273-3f48064c3a98","Type":"ContainerStarted","Data":"47bc0f16f05b93bd8c57a366b89d2baf0be82577fe82cd9fcb3db0b3517142c4"}
Oct 30 05:03:46.022343 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:46.022293    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-0b21ba/distinct-w5-0b21ba-kserve-7674c59666-m2sbd" podStartSLOduration=4.02227007 podStartE2EDuration="4.02227007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:03:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:03:46.017758477 +0000 UTC m=+1155.241578391" watchObservedRunningTime="2026-10-30 05:03:46.02227007 +0000 UTC m=+1155.246089968"
Oct 30 05:03:46.314265 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:46.314183    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:46.372540 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:46.372511    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0b21ba/distinct-w5-0b21ba-kserve-7674c59666-m2sbd"
Oct 30 05:03:46.570861 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:46.570787    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:47.791090 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:47.791057    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:47.893842 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:47.893812    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:48.108845 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:48.108814    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:48.208698 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:48.208663    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:48.719928 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:48.719874    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:49.866632 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:49.866570    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:51.566819 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:51.566783    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:53.080842 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:53.080797    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-324e5f78-rtx7p"]
Oct 30 05:03:53.081339 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:53.080866    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-324e5f78-rtx7p"]
Oct 30 05:03:53.372446 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:53.372363    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-0b21ba/distinct-w5-0b21ba-kserve-7674c59666-m2sbd"
Oct 30 05:03:53.385668 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:53.385644    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-0b21ba/distinct-w5-0b21ba-kserve-7674c59666-m2sbd"
Oct 30 05:03:53.503975 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:53.503941    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:54.089922 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:54.089873    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0b21ba/distinct-w5-0b21ba-kserve-7674c59666-m2sbd"
Oct 30 05:03:54.258142 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:54.258111    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:54.362512 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:54.362437    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:54.891363 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:54.891331    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4a8c37c-87a3-454d-a98d-db82229dc871" path="/var/lib/kubelet/pods/f4a8c37c-87a3-454d-a98d-db82229dc871/volumes"
Oct 30 05:03:55.490935 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:55.490874    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:55.587171 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:55.587138    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:55.965548 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:55.965516    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:56.207607 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:56.207560    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:56.820842 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:03:56.820804    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:03:57.452330 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:57.452285    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:59.267605 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:59.267574    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:59.798145 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:59.798093    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:03:59.895825 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:03:59.895791    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:01.031507 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:01.031473    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-521287b6-tgxzz"]
Oct 30 05:04:01.043343 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:01.043316    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-521287b6-tgxzz"]
Oct 30 05:04:01.258076 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:01.258042    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:01.491421 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:01.491390    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:01.590098 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:01.590059    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:02.892792 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:02.892757    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="336ce57a-6ab1-41bd-9c74-6ba31bd6e72b" path="/var/lib/kubelet/pods/336ce57a-6ab1-41bd-9c74-6ba31bd6e72b/volumes"
Oct 30 05:04:03.333750 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:03.333721    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:05.365129 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:05.365075    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:08.813934 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:08.813063    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849" podUID="a9707f50-d5e1-409f-b8f3-c58629677aaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:04:08.972260 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:08.972214    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9" podUID="1647f909-5580-4219-a2e1-30b4cd994379" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:04:09.000018 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:08.998446    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6" podUID="47ff272a-ac5c-4f59-bf7b-5c0692d09386" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:04:09.156927 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:09.156112    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5" podUID="ebf8f4a2-8ac6-48ec-bdba-3f6a99653264" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 05:04:09.156927 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:09.156368    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb" podUID="6f16bd6a-a7f0-49bb-844f-57b777bde2ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 05:04:09.208946 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:09.202007    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58" podUID="c27f3b21-1447-4af9-8d99-c04fa9b23cda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:04:09.231943 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:09.231801    5898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5" podUID="ebf8f4a2-8ac6-48ec-bdba-3f6a99653264" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:04:09.932635 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:09.932593    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d67de78b-c65b577cb-q4q8b"]
Oct 30 05:04:10.324982 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:10.299301    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:10.502090 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:10.497279    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:10.856640 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:10.856608    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:11.335954 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:11.335813    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:13.341363 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:13.341325    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-0b21ba/distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6"]
Oct 30 05:04:13.346028 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:13.346001    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-0b21ba/distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6"
Oct 30 05:04:13.365827 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:13.365796    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-0b21ba/distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6"]
Oct 30 05:04:13.370166 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:13.370125    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-0b21ba\"/\"distinct-2-w5-0b21ba-kserve-self-signed-certs\""
Oct 30 05:04:13.386231 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:13.386180    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:13.421495 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:13.421464    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d30a7a86-f1da-451c-81dd-ebdbaeb4f8af-home\") pod \"distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6\" (UID: \"d30a7a86-f1da-451c-81dd-ebdbaeb4f8af\") " pod="e2e-models-e2e-worker-w5-0b21ba/distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6"
Oct 30 05:04:13.421660 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:13.421531    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d30a7a86-f1da-451c-81dd-ebdbaeb4f8af-model-cache\") pod \"distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6\" (UID: \"d30a7a86-f1da-451c-81dd-ebdbaeb4f8af\") " pod="e2e-models-e2e-worker-w5-0b21ba/distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6"
Oct 30 05:04:13.421725 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:13.421662    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d30a7a86-f1da-451c-81dd-ebdbaeb4f8af-tmp-dir\") pod \"distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6\" (UID: \"d30a7a86-f1da-451c-81dd-ebdbaeb4f8af\") " pod="e2e-models-e2e-worker-w5-0b21ba/distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6"
Oct 30 05:04:13.421784 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:13.421743    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2qrr\" (UniqueName: \"kubernetes.io/projected/d30a7a86-f1da-451c-81dd-ebdbaeb4f8af-kube-api-access-s2qrr\") pod \"distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6\" (UID: \"d30a7a86-f1da-451c-81dd-ebdbaeb4f8af\") " pod="e2e-models-e2e-worker-w5-0b21ba/distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6"
Oct 30 05:04:13.421784 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:13.421764    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d30a7a86-f1da-451c-81dd-ebdbaeb4f8af-tls-certs\") pod \"distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6\" (UID: \"d30a7a86-f1da-451c-81dd-ebdbaeb4f8af\") " pod="e2e-models-e2e-worker-w5-0b21ba/distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6"
Oct 30 05:04:13.421913 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:13.421838    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d30a7a86-f1da-451c-81dd-ebdbaeb4f8af-dshm\") pod \"distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6\" (UID: \"d30a7a86-f1da-451c-81dd-ebdbaeb4f8af\") " pod="e2e-models-e2e-worker-w5-0b21ba/distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6"
Oct 30 05:04:13.523099 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:13.523061    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d30a7a86-f1da-451c-81dd-ebdbaeb4f8af-dshm\") pod \"distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6\" (UID: \"d30a7a86-f1da-451c-81dd-ebdbaeb4f8af\") " pod="e2e-models-e2e-worker-w5-0b21ba/distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6"
Oct 30 05:04:13.523283 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:13.523222    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d30a7a86-f1da-451c-81dd-ebdbaeb4f8af-home\") pod \"distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6\" (UID: \"d30a7a86-f1da-451c-81dd-ebdbaeb4f8af\") " pod="e2e-models-e2e-worker-w5-0b21ba/distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6"
Oct 30 05:04:13.523283 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:13.523269    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d30a7a86-f1da-451c-81dd-ebdbaeb4f8af-model-cache\") pod \"distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6\" (UID: \"d30a7a86-f1da-451c-81dd-ebdbaeb4f8af\") " pod="e2e-models-e2e-worker-w5-0b21ba/distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6"
Oct 30 05:04:13.523410 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:13.523320    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d30a7a86-f1da-451c-81dd-ebdbaeb4f8af-tmp-dir\") pod \"distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6\" (UID: \"d30a7a86-f1da-451c-81dd-ebdbaeb4f8af\") " pod="e2e-models-e2e-worker-w5-0b21ba/distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6"
Oct 30 05:04:13.523537 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:13.523405    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2qrr\" (UniqueName: \"kubernetes.io/projected/d30a7a86-f1da-451c-81dd-ebdbaeb4f8af-kube-api-access-s2qrr\") pod \"distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6\" (UID: \"d30a7a86-f1da-451c-81dd-ebdbaeb4f8af\") " pod="e2e-models-e2e-worker-w5-0b21ba/distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6"
Oct 30 05:04:13.523537 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:13.523433    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d30a7a86-f1da-451c-81dd-ebdbaeb4f8af-tls-certs\") pod \"distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6\" (UID: \"d30a7a86-f1da-451c-81dd-ebdbaeb4f8af\") " pod="e2e-models-e2e-worker-w5-0b21ba/distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6"
Oct 30 05:04:13.523697 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:13.523672    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d30a7a86-f1da-451c-81dd-ebdbaeb4f8af-home\") pod \"distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6\" (UID: \"d30a7a86-f1da-451c-81dd-ebdbaeb4f8af\") " pod="e2e-models-e2e-worker-w5-0b21ba/distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6"
Oct 30 05:04:13.523815 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:13.523760    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d30a7a86-f1da-451c-81dd-ebdbaeb4f8af-tmp-dir\") pod \"distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6\" (UID: \"d30a7a86-f1da-451c-81dd-ebdbaeb4f8af\") " pod="e2e-models-e2e-worker-w5-0b21ba/distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6"
Oct 30 05:04:13.524015 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:13.523988    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d30a7a86-f1da-451c-81dd-ebdbaeb4f8af-model-cache\") pod \"distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6\" (UID: \"d30a7a86-f1da-451c-81dd-ebdbaeb4f8af\") " pod="e2e-models-e2e-worker-w5-0b21ba/distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6"
Oct 30 05:04:13.525855 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:13.525822    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d30a7a86-f1da-451c-81dd-ebdbaeb4f8af-dshm\") pod \"distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6\" (UID: \"d30a7a86-f1da-451c-81dd-ebdbaeb4f8af\") " pod="e2e-models-e2e-worker-w5-0b21ba/distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6"
Oct 30 05:04:13.526098 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:13.526074    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d30a7a86-f1da-451c-81dd-ebdbaeb4f8af-tls-certs\") pod \"distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6\" (UID: \"d30a7a86-f1da-451c-81dd-ebdbaeb4f8af\") " pod="e2e-models-e2e-worker-w5-0b21ba/distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6"
Oct 30 05:04:13.543156 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:13.543128    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2qrr\" (UniqueName: \"kubernetes.io/projected/d30a7a86-f1da-451c-81dd-ebdbaeb4f8af-kube-api-access-s2qrr\") pod \"distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6\" (UID: \"d30a7a86-f1da-451c-81dd-ebdbaeb4f8af\") " pod="e2e-models-e2e-worker-w5-0b21ba/distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6"
Oct 30 05:04:13.664845 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:13.664508    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-0b21ba/distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6"
Oct 30 05:04:13.983061 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:04:13.982890    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd30a7a86_f1da_451c_81dd_ebdbaeb4f8af.slice/crio-cf0111a03ec810d6f13daec0155e02b31358852d892538ca4b8bd24a9959e6b1 WatchSource:0}: Error finding container cf0111a03ec810d6f13daec0155e02b31358852d892538ca4b8bd24a9959e6b1: Status 404 returned error can't find the container with id cf0111a03ec810d6f13daec0155e02b31358852d892538ca4b8bd24a9959e6b1
Oct 30 05:04:13.984550 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:13.984525    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-0b21ba/distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6"]
Oct 30 05:04:14.713884 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:14.713856    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:14.899582 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:14.896615    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-0b21ba/distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6" event={"ID":"d30a7a86-f1da-451c-81dd-ebdbaeb4f8af","Type":"ContainerStarted","Data":"3e50dbcf9a7bdcde2a0c56f382bad3a52e565f5ffed875946cf4f38c2755d4cd"}
Oct 30 05:04:14.899582 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:14.896674    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-0b21ba/distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6" event={"ID":"d30a7a86-f1da-451c-81dd-ebdbaeb4f8af","Type":"ContainerStarted","Data":"cf0111a03ec810d6f13daec0155e02b31358852d892538ca4b8bd24a9959e6b1"}
Oct 30 05:04:14.924712 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:14.924660    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-0b21ba/distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6" podStartSLOduration=1.92464711 podStartE2EDuration="1.92464711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:04:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:04:14.922608022 +0000 UTC m=+1184.146427942" watchObservedRunningTime="2026-10-30 05:04:14.92464711 +0000 UTC m=+1184.148467010"
Oct 30 05:04:14.933864 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:14.933812    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-d67de78b-c65b577cb-q4q8b" podUID="36521c09-d623-48e2-801a-e3be3e2e25fa" containerName="payload-processing" containerID="cri-o://da87e7b36567b4ec059b377c3433acbcd98fd5651305015d1d32bb8dfcb8a039" gracePeriod=25
Oct 30 05:04:15.218291 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:15.218263    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d67de78b-c65b577cb-q4q8b"
Oct 30 05:04:15.351154 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:15.351123    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36521c09-d623-48e2-801a-e3be3e2e25fa-kube-api-access-2dk9z\" (UniqueName: \"kubernetes.io/projected/36521c09-d623-48e2-801a-e3be3e2e25fa-kube-api-access-2dk9z\") pod \"36521c09-d623-48e2-801a-e3be3e2e25fa\" (UID: \"36521c09-d623-48e2-801a-e3be3e2e25fa\") "
Oct 30 05:04:15.351375 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:15.351362    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36521c09-d623-48e2-801a-e3be3e2e25fa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36521c09-d623-48e2-801a-e3be3e2e25fa-plugins-config-volume\") pod \"36521c09-d623-48e2-801a-e3be3e2e25fa\" (UID: \"36521c09-d623-48e2-801a-e3be3e2e25fa\") "
Oct 30 05:04:15.351751 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:15.351726    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36521c09-d623-48e2-801a-e3be3e2e25fa-plugins-config-volume" pod "36521c09-d623-48e2-801a-e3be3e2e25fa" (UID: "36521c09-d623-48e2-801a-e3be3e2e25fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:04:15.351875 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:15.351862    5898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36521c09-d623-48e2-801a-e3be3e2e25fa-plugins-config-volume\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:04:15.353436 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:15.353413    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36521c09-d623-48e2-801a-e3be3e2e25fa-kube-api-access-2dk9z" pod "36521c09-d623-48e2-801a-e3be3e2e25fa" (UID: "36521c09-d623-48e2-801a-e3be3e2e25fa"). InnerVolumeSpecName "kube-api-access-2dk9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:04:15.453117 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:15.453083    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dk9z\" (UniqueName: \"kubernetes.io/projected/36521c09-d623-48e2-801a-e3be3e2e25fa-kube-api-access-2dk9z\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:04:15.664799 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:15.664725    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0b21ba/distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6"
Oct 30 05:04:15.891392 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:15.891356    5898 generic.go:356] "Generic (PLEG): container finished" podID="36521c09-d623-48e2-801a-e3be3e2e25fa" containerID="da87e7b36567b4ec059b377c3433acbcd98fd5651305015d1d32bb8dfcb8a039" exitCode=0
Oct 30 05:04:15.891890 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:15.891397    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d67de78b-c65b577cb-q4q8b"
Oct 30 05:04:15.891890 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:15.891438    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d67de78b-c65b577cb-q4q8b" event={"ID":"36521c09-d623-48e2-801a-e3be3e2e25fa","Type":"ContainerDied","Data":"da87e7b36567b4ec059b377c3433acbcd98fd5651305015d1d32bb8dfcb8a039"}
Oct 30 05:04:15.891890 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:15.891490    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d67de78b-c65b577cb-q4q8b" event={"ID":"36521c09-d623-48e2-801a-e3be3e2e25fa","Type":"ContainerDied","Data":"0dddedcd9afba6545e69ef5083603bed97d40ffef3c6e5f69eb20a1462a954fe"}
Oct 30 05:04:15.891890 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:15.891518    5898 scope.go:122] "RemoveContainer" containerID="da87e7b36567b4ec059b377c3433acbcd98fd5651305015d1d32bb8dfcb8a039"
Oct 30 05:04:15.902440 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:15.902420    5898 scope.go:122] "RemoveContainer" containerID="da87e7b36567b4ec059b377c3433acbcd98fd5651305015d1d32bb8dfcb8a039"
Oct 30 05:04:15.902692 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:04:15.902671    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da87e7b36567b4ec059b377c3433acbcd98fd5651305015d1d32bb8dfcb8a039\": container with ID starting with da87e7b36567b4ec059b377c3433acbcd98fd5651305015d1d32bb8dfcb8a039 not found: ID does not exist" containerID="da87e7b36567b4ec059b377c3433acbcd98fd5651305015d1d32bb8dfcb8a039"
Oct 30 05:04:15.902764 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:15.902706    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da87e7b36567b4ec059b377c3433acbcd98fd5651305015d1d32bb8dfcb8a039"} err="failed to get container status \"da87e7b36567b4ec059b377c3433acbcd98fd5651305015d1d32bb8dfcb8a039\": rpc error: code = NotFound desc = could not find container \"da87e7b36567b4ec059b377c3433acbcd98fd5651305015d1d32bb8dfcb8a039\": container with ID starting with da87e7b36567b4ec059b377c3433acbcd98fd5651305015d1d32bb8dfcb8a039 not found: ID does not exist"
Oct 30 05:04:15.942408 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:15.942340    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d67de78b-c65b577cb-q4q8b"]
Oct 30 05:04:15.956767 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:15.956738    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d67de78b-c65b577cb-q4q8b"]
Oct 30 05:04:16.371977 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:16.371945    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d67de78b-67f5fbb5c5-dp59d"]
Oct 30 05:04:16.372469 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:16.372214    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d67de78b-67f5fbb5c5-dp59d" podUID="92cd0f2e-f7e5-47d3-84e7-aa96c8d75736" containerName="maas-api" containerID="cri-o://931194ac5f53a2d20dec2b97e762479428867976615a0b65efb38fbb427e5a4d" gracePeriod=30
Oct 30 05:04:16.416644 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:16.416618    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:16.609673 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:16.609631    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b0fba292-65679f777d-tkn74"]
Oct 30 05:04:16.705964 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:16.705933    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d67de78b-67f5fbb5c5-dp59d"
Oct 30 05:04:16.719557 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:16.717162    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:16.815466 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:16.815433    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:16.891980 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:16.891951    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36521c09-d623-48e2-801a-e3be3e2e25fa" path="/var/lib/kubelet/pods/36521c09-d623-48e2-801a-e3be3e2e25fa/volumes"
Oct 30 05:04:16.895924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:16.895885    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92cd0f2e-f7e5-47d3-84e7-aa96c8d75736-kube-api-access-c79b9\" (UniqueName: \"kubernetes.io/projected/92cd0f2e-f7e5-47d3-84e7-aa96c8d75736-kube-api-access-c79b9\") pod \"92cd0f2e-f7e5-47d3-84e7-aa96c8d75736\" (UID: \"92cd0f2e-f7e5-47d3-84e7-aa96c8d75736\") "
Oct 30 05:04:16.896073 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:16.895950    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92cd0f2e-f7e5-47d3-84e7-aa96c8d75736-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92cd0f2e-f7e5-47d3-84e7-aa96c8d75736-openshift-service-ca\") pod \"92cd0f2e-f7e5-47d3-84e7-aa96c8d75736\" (UID: \"92cd0f2e-f7e5-47d3-84e7-aa96c8d75736\") "
Oct 30 05:04:16.896073 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:16.895995    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92cd0f2e-f7e5-47d3-84e7-aa96c8d75736-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92cd0f2e-f7e5-47d3-84e7-aa96c8d75736-maas-api-tls\") pod \"92cd0f2e-f7e5-47d3-84e7-aa96c8d75736\" (UID: \"92cd0f2e-f7e5-47d3-84e7-aa96c8d75736\") "
Oct 30 05:04:16.896239 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:16.896215    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92cd0f2e-f7e5-47d3-84e7-aa96c8d75736-metrics-certs\" (UniqueName: \"kubernetes.io/secret/92cd0f2e-f7e5-47d3-84e7-aa96c8d75736-metrics-certs\") pod \"92cd0f2e-f7e5-47d3-84e7-aa96c8d75736\" (UID: \"92cd0f2e-f7e5-47d3-84e7-aa96c8d75736\") "
Oct 30 05:04:16.896375 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:16.896350    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92cd0f2e-f7e5-47d3-84e7-aa96c8d75736-openshift-service-ca" pod "92cd0f2e-f7e5-47d3-84e7-aa96c8d75736" (UID: "92cd0f2e-f7e5-47d3-84e7-aa96c8d75736"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:04:16.896865 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:16.896837    5898 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92cd0f2e-f7e5-47d3-84e7-aa96c8d75736-openshift-service-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:04:16.898586 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:16.898560    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92cd0f2e-f7e5-47d3-84e7-aa96c8d75736-kube-api-access-c79b9" pod "92cd0f2e-f7e5-47d3-84e7-aa96c8d75736" (UID: "92cd0f2e-f7e5-47d3-84e7-aa96c8d75736"). InnerVolumeSpecName "kube-api-access-c79b9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:04:16.899235 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:16.899197    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92cd0f2e-f7e5-47d3-84e7-aa96c8d75736-metrics-certs" pod "92cd0f2e-f7e5-47d3-84e7-aa96c8d75736" (UID: "92cd0f2e-f7e5-47d3-84e7-aa96c8d75736"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:04:16.899427 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:16.899410    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92cd0f2e-f7e5-47d3-84e7-aa96c8d75736-maas-api-tls" pod "92cd0f2e-f7e5-47d3-84e7-aa96c8d75736" (UID: "92cd0f2e-f7e5-47d3-84e7-aa96c8d75736"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:04:16.902130 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:16.902102    5898 generic.go:356] "Generic (PLEG): container finished" podID="92cd0f2e-f7e5-47d3-84e7-aa96c8d75736" containerID="931194ac5f53a2d20dec2b97e762479428867976615a0b65efb38fbb427e5a4d" exitCode=0
Oct 30 05:04:16.902299 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:16.902280    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d67de78b-67f5fbb5c5-dp59d" event={"ID":"92cd0f2e-f7e5-47d3-84e7-aa96c8d75736","Type":"ContainerDied","Data":"931194ac5f53a2d20dec2b97e762479428867976615a0b65efb38fbb427e5a4d"}
Oct 30 05:04:16.902374 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:16.902332    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d67de78b-67f5fbb5c5-dp59d" event={"ID":"92cd0f2e-f7e5-47d3-84e7-aa96c8d75736","Type":"ContainerDied","Data":"7f2cbffd93c21fb3fa83d45a6bd79c1769f16140ea50ec3c47fa65f388f63633"}
Oct 30 05:04:16.902374 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:16.902362    5898 scope.go:122] "RemoveContainer" containerID="931194ac5f53a2d20dec2b97e762479428867976615a0b65efb38fbb427e5a4d"
Oct 30 05:04:16.902499 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:16.902485    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d67de78b-67f5fbb5c5-dp59d"
Oct 30 05:04:16.935975 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:16.935952    5898 scope.go:122] "RemoveContainer" containerID="931194ac5f53a2d20dec2b97e762479428867976615a0b65efb38fbb427e5a4d"
Oct 30 05:04:16.936408 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:04:16.936348    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"931194ac5f53a2d20dec2b97e762479428867976615a0b65efb38fbb427e5a4d\": container with ID starting with 931194ac5f53a2d20dec2b97e762479428867976615a0b65efb38fbb427e5a4d not found: ID does not exist" containerID="931194ac5f53a2d20dec2b97e762479428867976615a0b65efb38fbb427e5a4d"
Oct 30 05:04:16.936516 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:16.936417    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"931194ac5f53a2d20dec2b97e762479428867976615a0b65efb38fbb427e5a4d"} err="failed to get container status \"931194ac5f53a2d20dec2b97e762479428867976615a0b65efb38fbb427e5a4d\": rpc error: code = NotFound desc = could not find container \"931194ac5f53a2d20dec2b97e762479428867976615a0b65efb38fbb427e5a4d\": container with ID starting with 931194ac5f53a2d20dec2b97e762479428867976615a0b65efb38fbb427e5a4d not found: ID does not exist"
Oct 30 05:04:17.018018 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:17.006043    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c79b9\" (UniqueName: \"kubernetes.io/projected/92cd0f2e-f7e5-47d3-84e7-aa96c8d75736-kube-api-access-c79b9\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:04:17.018018 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:17.006076    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92cd0f2e-f7e5-47d3-84e7-aa96c8d75736-maas-api-tls\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:04:17.018018 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:17.006092    5898 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92cd0f2e-f7e5-47d3-84e7-aa96c8d75736-metrics-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:04:17.018018 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:17.015284    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d67de78b-67f5fbb5c5-dp59d"]
Oct 30 05:04:17.018018 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:17.015737    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:17.018417 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:17.018281    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d67de78b-67f5fbb5c5-dp59d"]
Oct 30 05:04:17.273873 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:17.273787    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:17.377621 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:17.377591    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:17.645771 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:17.645732    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:18.283190 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:18.283158    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:18.398299 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:18.398270    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:18.568814 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:18.568738    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:18.892048 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:18.892016    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92cd0f2e-f7e5-47d3-84e7-aa96c8d75736" path="/var/lib/kubelet/pods/92cd0f2e-f7e5-47d3-84e7-aa96c8d75736/volumes"
Oct 30 05:04:18.994731 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:18.994693    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk"]
Oct 30 05:04:18.995510 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:18.995485    5898 cpu_manager.go:414] "containerMap: removing container" podUID="36521c09-d623-48e2-801a-e3be3e2e25fa" containerName="payload-processing"
Oct 30 05:04:18.995510 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:18.995502    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36521c09-d623-48e2-801a-e3be3e2e25fa" containerName="payload-processing"
Oct 30 05:04:18.995742 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:18.995517    5898 cpu_manager.go:414] "containerMap: removing container" podUID="92cd0f2e-f7e5-47d3-84e7-aa96c8d75736" containerName="maas-api"
Oct 30 05:04:18.995742 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:18.995526    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92cd0f2e-f7e5-47d3-84e7-aa96c8d75736" containerName="maas-api"
Oct 30 05:04:18.995742 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:18.995651    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="36521c09-d623-48e2-801a-e3be3e2e25fa" containerName="payload-processing"
Oct 30 05:04:18.995742 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:18.995661    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="92cd0f2e-f7e5-47d3-84e7-aa96c8d75736" containerName="maas-api"
Oct 30 05:04:18.999591 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:18.999565    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk"
Oct 30 05:04:19.039923 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.029269    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk"]
Oct 30 05:04:19.039923 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.029427    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-5a926a9d-gw-openshift-default-dockercfg-p7l8x\""
Oct 30 05:04:19.053568 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.053534    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:19.065760 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.065721    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52273f50-a59c-4f49-88cc-1e12c205713a-workload-certs\") pod \"e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk\" (UID: \"52273f50-a59c-4f49-88cc-1e12c205713a\") " pod="openshift-ingress/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk"
Oct 30 05:04:19.066104 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.065892    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52273f50-a59c-4f49-88cc-1e12c205713a-credential-socket\") pod \"e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk\" (UID: \"52273f50-a59c-4f49-88cc-1e12c205713a\") " pod="openshift-ingress/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk"
Oct 30 05:04:19.066104 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.065991    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glnsn\" (UniqueName: \"kubernetes.io/projected/52273f50-a59c-4f49-88cc-1e12c205713a-kube-api-access-glnsn\") pod \"e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk\" (UID: \"52273f50-a59c-4f49-88cc-1e12c205713a\") " pod="openshift-ingress/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk"
Oct 30 05:04:19.066104 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.066023    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52273f50-a59c-4f49-88cc-1e12c205713a-workload-socket\") pod \"e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk\" (UID: \"52273f50-a59c-4f49-88cc-1e12c205713a\") " pod="openshift-ingress/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk"
Oct 30 05:04:19.066104 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.066076    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52273f50-a59c-4f49-88cc-1e12c205713a-istiod-ca-cert\") pod \"e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk\" (UID: \"52273f50-a59c-4f49-88cc-1e12c205713a\") " pod="openshift-ingress/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk"
Oct 30 05:04:19.066304 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.066221    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52273f50-a59c-4f49-88cc-1e12c205713a-istio-data\") pod \"e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk\" (UID: \"52273f50-a59c-4f49-88cc-1e12c205713a\") " pod="openshift-ingress/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk"
Oct 30 05:04:19.066304 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.066281    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52273f50-a59c-4f49-88cc-1e12c205713a-istio-envoy\") pod \"e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk\" (UID: \"52273f50-a59c-4f49-88cc-1e12c205713a\") " pod="openshift-ingress/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk"
Oct 30 05:04:19.066380 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.066338    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52273f50-a59c-4f49-88cc-1e12c205713a-istio-token\") pod \"e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk\" (UID: \"52273f50-a59c-4f49-88cc-1e12c205713a\") " pod="openshift-ingress/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk"
Oct 30 05:04:19.066422 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.066391    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52273f50-a59c-4f49-88cc-1e12c205713a-istio-podinfo\") pod \"e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk\" (UID: \"52273f50-a59c-4f49-88cc-1e12c205713a\") " pod="openshift-ingress/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk"
Oct 30 05:04:19.152277 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.152202    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:19.167815 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.167789    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52273f50-a59c-4f49-88cc-1e12c205713a-credential-socket\") pod \"e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk\" (UID: \"52273f50-a59c-4f49-88cc-1e12c205713a\") " pod="openshift-ingress/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk"
Oct 30 05:04:19.167990 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.167831    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glnsn\" (UniqueName: \"kubernetes.io/projected/52273f50-a59c-4f49-88cc-1e12c205713a-kube-api-access-glnsn\") pod \"e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk\" (UID: \"52273f50-a59c-4f49-88cc-1e12c205713a\") " pod="openshift-ingress/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk"
Oct 30 05:04:19.167990 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.167850    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52273f50-a59c-4f49-88cc-1e12c205713a-workload-socket\") pod \"e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk\" (UID: \"52273f50-a59c-4f49-88cc-1e12c205713a\") " pod="openshift-ingress/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk"
Oct 30 05:04:19.167990 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.167884    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52273f50-a59c-4f49-88cc-1e12c205713a-istiod-ca-cert\") pod \"e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk\" (UID: \"52273f50-a59c-4f49-88cc-1e12c205713a\") " pod="openshift-ingress/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk"
Oct 30 05:04:19.167990 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.167967    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52273f50-a59c-4f49-88cc-1e12c205713a-istio-data\") pod \"e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk\" (UID: \"52273f50-a59c-4f49-88cc-1e12c205713a\") " pod="openshift-ingress/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk"
Oct 30 05:04:19.168213 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.168022    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52273f50-a59c-4f49-88cc-1e12c205713a-istio-envoy\") pod \"e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk\" (UID: \"52273f50-a59c-4f49-88cc-1e12c205713a\") " pod="openshift-ingress/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk"
Oct 30 05:04:19.168213 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.168088    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52273f50-a59c-4f49-88cc-1e12c205713a-istio-token\") pod \"e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk\" (UID: \"52273f50-a59c-4f49-88cc-1e12c205713a\") " pod="openshift-ingress/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk"
Oct 30 05:04:19.168213 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.168147    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52273f50-a59c-4f49-88cc-1e12c205713a-istio-podinfo\") pod \"e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk\" (UID: \"52273f50-a59c-4f49-88cc-1e12c205713a\") " pod="openshift-ingress/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk"
Oct 30 05:04:19.168360 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.168258    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52273f50-a59c-4f49-88cc-1e12c205713a-credential-socket\") pod \"e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk\" (UID: \"52273f50-a59c-4f49-88cc-1e12c205713a\") " pod="openshift-ingress/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk"
Oct 30 05:04:19.168360 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.168317    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52273f50-a59c-4f49-88cc-1e12c205713a-workload-socket\") pod \"e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk\" (UID: \"52273f50-a59c-4f49-88cc-1e12c205713a\") " pod="openshift-ingress/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk"
Oct 30 05:04:19.168360 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.168351    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52273f50-a59c-4f49-88cc-1e12c205713a-workload-certs\") pod \"e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk\" (UID: \"52273f50-a59c-4f49-88cc-1e12c205713a\") " pod="openshift-ingress/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk"
Oct 30 05:04:19.168654 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.168624    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52273f50-a59c-4f49-88cc-1e12c205713a-istiod-ca-cert\") pod \"e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk\" (UID: \"52273f50-a59c-4f49-88cc-1e12c205713a\") " pod="openshift-ingress/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk"
Oct 30 05:04:19.168764 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.168716    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52273f50-a59c-4f49-88cc-1e12c205713a-workload-certs\") pod \"e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk\" (UID: \"52273f50-a59c-4f49-88cc-1e12c205713a\") " pod="openshift-ingress/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk"
Oct 30 05:04:19.168825 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.168790    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52273f50-a59c-4f49-88cc-1e12c205713a-istio-data\") pod \"e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk\" (UID: \"52273f50-a59c-4f49-88cc-1e12c205713a\") " pod="openshift-ingress/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk"
Oct 30 05:04:19.170470 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.170445    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52273f50-a59c-4f49-88cc-1e12c205713a-istio-envoy\") pod \"e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk\" (UID: \"52273f50-a59c-4f49-88cc-1e12c205713a\") " pod="openshift-ingress/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk"
Oct 30 05:04:19.170686 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.170667    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52273f50-a59c-4f49-88cc-1e12c205713a-istio-podinfo\") pod \"e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk\" (UID: \"52273f50-a59c-4f49-88cc-1e12c205713a\") " pod="openshift-ingress/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk"
Oct 30 05:04:19.186414 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.186390    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glnsn\" (UniqueName: \"kubernetes.io/projected/52273f50-a59c-4f49-88cc-1e12c205713a-kube-api-access-glnsn\") pod \"e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk\" (UID: \"52273f50-a59c-4f49-88cc-1e12c205713a\") " pod="openshift-ingress/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk"
Oct 30 05:04:19.187096 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.187071    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52273f50-a59c-4f49-88cc-1e12c205713a-istio-token\") pod \"e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk\" (UID: \"52273f50-a59c-4f49-88cc-1e12c205713a\") " pod="openshift-ingress/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk"
Oct 30 05:04:19.280384 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.280345    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ft7s9"]
Oct 30 05:04:19.287602 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.287575    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ft7s9"
Oct 30 05:04:19.309226 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.308962    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 05:04:19.309226 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.309032    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 05:04:19.309226 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.309079    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 05:04:19.309226 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.309197    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-8lhdl\""
Oct 30 05:04:19.318374 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.317234    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk"
Oct 30 05:04:19.330621 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.326058    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ft7s9"]
Oct 30 05:04:19.373927 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.373870    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f7b35ef1-25f4-46ac-9a62-fe5e2e027af4-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ft7s9\" (UID: \"f7b35ef1-25f4-46ac-9a62-fe5e2e027af4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ft7s9"
Oct 30 05:04:19.374081 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.373947    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f7b35ef1-25f4-46ac-9a62-fe5e2e027af4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ft7s9\" (UID: \"f7b35ef1-25f4-46ac-9a62-fe5e2e027af4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ft7s9"
Oct 30 05:04:19.374081 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.374042    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7b35ef1-25f4-46ac-9a62-fe5e2e027af4-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ft7s9\" (UID: \"f7b35ef1-25f4-46ac-9a62-fe5e2e027af4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ft7s9"
Oct 30 05:04:19.374177 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.374078    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7b35ef1-25f4-46ac-9a62-fe5e2e027af4-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ft7s9\" (UID: \"f7b35ef1-25f4-46ac-9a62-fe5e2e027af4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ft7s9"
Oct 30 05:04:19.374177 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.374102    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f7b35ef1-25f4-46ac-9a62-fe5e2e027af4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ft7s9\" (UID: \"f7b35ef1-25f4-46ac-9a62-fe5e2e027af4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ft7s9"
Oct 30 05:04:19.374355 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.374334    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx4n6\" (UniqueName: \"kubernetes.io/projected/f7b35ef1-25f4-46ac-9a62-fe5e2e027af4-kube-api-access-qx4n6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ft7s9\" (UID: \"f7b35ef1-25f4-46ac-9a62-fe5e2e027af4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ft7s9"
Oct 30 05:04:19.374419 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.374381    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f7b35ef1-25f4-46ac-9a62-fe5e2e027af4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ft7s9\" (UID: \"f7b35ef1-25f4-46ac-9a62-fe5e2e027af4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ft7s9"
Oct 30 05:04:19.494256 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.490719    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx4n6\" (UniqueName: \"kubernetes.io/projected/f7b35ef1-25f4-46ac-9a62-fe5e2e027af4-kube-api-access-qx4n6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ft7s9\" (UID: \"f7b35ef1-25f4-46ac-9a62-fe5e2e027af4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ft7s9"
Oct 30 05:04:19.494256 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.490794    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f7b35ef1-25f4-46ac-9a62-fe5e2e027af4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ft7s9\" (UID: \"f7b35ef1-25f4-46ac-9a62-fe5e2e027af4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ft7s9"
Oct 30 05:04:19.494256 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.490939    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f7b35ef1-25f4-46ac-9a62-fe5e2e027af4-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ft7s9\" (UID: \"f7b35ef1-25f4-46ac-9a62-fe5e2e027af4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ft7s9"
Oct 30 05:04:19.494256 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.490972    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f7b35ef1-25f4-46ac-9a62-fe5e2e027af4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ft7s9\" (UID: \"f7b35ef1-25f4-46ac-9a62-fe5e2e027af4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ft7s9"
Oct 30 05:04:19.494256 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.491085    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7b35ef1-25f4-46ac-9a62-fe5e2e027af4-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ft7s9\" (UID: \"f7b35ef1-25f4-46ac-9a62-fe5e2e027af4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ft7s9"
Oct 30 05:04:19.494256 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.491121    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7b35ef1-25f4-46ac-9a62-fe5e2e027af4-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ft7s9\" (UID: \"f7b35ef1-25f4-46ac-9a62-fe5e2e027af4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ft7s9"
Oct 30 05:04:19.494256 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.491161    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f7b35ef1-25f4-46ac-9a62-fe5e2e027af4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ft7s9\" (UID: \"f7b35ef1-25f4-46ac-9a62-fe5e2e027af4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ft7s9"
Oct 30 05:04:19.494256 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.492392    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f7b35ef1-25f4-46ac-9a62-fe5e2e027af4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ft7s9\" (UID: \"f7b35ef1-25f4-46ac-9a62-fe5e2e027af4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ft7s9"
Oct 30 05:04:19.498118 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.498069    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f7b35ef1-25f4-46ac-9a62-fe5e2e027af4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ft7s9\" (UID: \"f7b35ef1-25f4-46ac-9a62-fe5e2e027af4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ft7s9"
Oct 30 05:04:19.511711 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.511216    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7b35ef1-25f4-46ac-9a62-fe5e2e027af4-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ft7s9\" (UID: \"f7b35ef1-25f4-46ac-9a62-fe5e2e027af4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ft7s9"
Oct 30 05:04:19.515515 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.515487    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f7b35ef1-25f4-46ac-9a62-fe5e2e027af4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ft7s9\" (UID: \"f7b35ef1-25f4-46ac-9a62-fe5e2e027af4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ft7s9"
Oct 30 05:04:19.517972 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.517771    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx4n6\" (UniqueName: \"kubernetes.io/projected/f7b35ef1-25f4-46ac-9a62-fe5e2e027af4-kube-api-access-qx4n6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ft7s9\" (UID: \"f7b35ef1-25f4-46ac-9a62-fe5e2e027af4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ft7s9"
Oct 30 05:04:19.518741 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.518631    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f7b35ef1-25f4-46ac-9a62-fe5e2e027af4-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ft7s9\" (UID: \"f7b35ef1-25f4-46ac-9a62-fe5e2e027af4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ft7s9"
Oct 30 05:04:19.543592 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.539715    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7b35ef1-25f4-46ac-9a62-fe5e2e027af4-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ft7s9\" (UID: \"f7b35ef1-25f4-46ac-9a62-fe5e2e027af4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ft7s9"
Oct 30 05:04:19.604427 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.604061    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ft7s9"
Oct 30 05:04:19.773681 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:04:19.773626    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52273f50_a59c_4f49_88cc_1e12c205713a.slice/crio-f2a8e22f7756a1511d8df0a72700fe6758f058638655d0b0523f08359f6a8f1f WatchSource:0}: Error finding container f2a8e22f7756a1511d8df0a72700fe6758f058638655d0b0523f08359f6a8f1f: Status 404 returned error can't find the container with id f2a8e22f7756a1511d8df0a72700fe6758f058638655d0b0523f08359f6a8f1f
Oct 30 05:04:19.775482 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.775448    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk"]
Oct 30 05:04:19.777485 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.777445    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:04:19.777592 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.777530    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:04:19.777592 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.777574    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:04:19.946089 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.946051    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk" event={"ID":"52273f50-a59c-4f49-88cc-1e12c205713a","Type":"ContainerStarted","Data":"9893bf2c15e3500895c24242fd14f0980bc4466805f522c1032b1d9aa6eee5df"}
Oct 30 05:04:19.946228 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.946118    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk" event={"ID":"52273f50-a59c-4f49-88cc-1e12c205713a","Type":"ContainerStarted","Data":"f2a8e22f7756a1511d8df0a72700fe6758f058638655d0b0523f08359f6a8f1f"}
Oct 30 05:04:19.948398 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:04:19.948372    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7b35ef1_25f4_46ac_9a62_fe5e2e027af4.slice/crio-be92d1ab6a6aec060e218f3a06d69d87f2e2e58a14761c99be9ba05a0a1bdd6f WatchSource:0}: Error finding container be92d1ab6a6aec060e218f3a06d69d87f2e2e58a14761c99be9ba05a0a1bdd6f: Status 404 returned error can't find the container with id be92d1ab6a6aec060e218f3a06d69d87f2e2e58a14761c99be9ba05a0a1bdd6f
Oct 30 05:04:19.952229 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.952208    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ft7s9"]
Oct 30 05:04:19.987712 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:19.987654    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk" podStartSLOduration=1.987630266 podStartE2EDuration="1.987630266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:04:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:04:19.986676724 +0000 UTC m=+1189.210496644" watchObservedRunningTime="2026-10-30 05:04:19.987630266 +0000 UTC m=+1189.211450168"
Oct 30 05:04:20.276853 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:20.276818    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:20.318130 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:20.318093    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk"
Oct 30 05:04:20.324402 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:20.324371    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk"
Oct 30 05:04:20.960530 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:20.960490    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ft7s9" event={"ID":"f7b35ef1-25f4-46ac-9a62-fe5e2e027af4","Type":"ContainerStarted","Data":"be92d1ab6a6aec060e218f3a06d69d87f2e2e58a14761c99be9ba05a0a1bdd6f"}
Oct 30 05:04:20.960717 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:20.960557    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk"
Oct 30 05:04:20.961416 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:20.961394    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk"
Oct 30 05:04:21.284318 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:21.284232    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:04:21.630976 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:21.630748    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-b0fba292-65679f777d-tkn74" podUID="afe14932-62ea-4859-95eb-f030280ac1e6" containerName="payload-processing" containerID="cri-o://b94100dba9aa5bbf8b6ee39d670d63c2f9b87d830b61d8b6192e99621bf335e1" gracePeriod=25
Oct 30 05:04:21.930998 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:21.930872    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:21.973274 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:21.973236    5898 generic.go:356] "Generic (PLEG): container finished" podID="afe14932-62ea-4859-95eb-f030280ac1e6" containerID="b94100dba9aa5bbf8b6ee39d670d63c2f9b87d830b61d8b6192e99621bf335e1" exitCode=0
Oct 30 05:04:21.973439 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:21.973310    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b0fba292-65679f777d-tkn74" event={"ID":"afe14932-62ea-4859-95eb-f030280ac1e6","Type":"ContainerDied","Data":"b94100dba9aa5bbf8b6ee39d670d63c2f9b87d830b61d8b6192e99621bf335e1"}
Oct 30 05:04:21.973439 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:21.973392    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b0fba292-65679f777d-tkn74" event={"ID":"afe14932-62ea-4859-95eb-f030280ac1e6","Type":"ContainerDied","Data":"9b8b9163a9170d4626c7ddc5da0391ee821df080863d4a4e5a39398c50f4fa2a"}
Oct 30 05:04:21.973439 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:21.973435    5898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b8b9163a9170d4626c7ddc5da0391ee821df080863d4a4e5a39398c50f4fa2a"
Oct 30 05:04:21.996416 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:21.996391    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b0fba292-65679f777d-tkn74"
Oct 30 05:04:22.015226 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:22.015196    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:22.152927 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:22.145654    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afe14932-62ea-4859-95eb-f030280ac1e6-kube-api-access-zr7v2\" (UniqueName: \"kubernetes.io/projected/afe14932-62ea-4859-95eb-f030280ac1e6-kube-api-access-zr7v2\") pod \"afe14932-62ea-4859-95eb-f030280ac1e6\" (UID: \"afe14932-62ea-4859-95eb-f030280ac1e6\") "
Oct 30 05:04:22.152927 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:22.145781    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afe14932-62ea-4859-95eb-f030280ac1e6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afe14932-62ea-4859-95eb-f030280ac1e6-plugins-config-volume\") pod \"afe14932-62ea-4859-95eb-f030280ac1e6\" (UID: \"afe14932-62ea-4859-95eb-f030280ac1e6\") "
Oct 30 05:04:22.152927 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:22.146183    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afe14932-62ea-4859-95eb-f030280ac1e6-plugins-config-volume" pod "afe14932-62ea-4859-95eb-f030280ac1e6" (UID: "afe14932-62ea-4859-95eb-f030280ac1e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:04:22.152927 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:22.146652    5898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afe14932-62ea-4859-95eb-f030280ac1e6-plugins-config-volume\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:04:22.152927 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:22.152428    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afe14932-62ea-4859-95eb-f030280ac1e6-kube-api-access-zr7v2" pod "afe14932-62ea-4859-95eb-f030280ac1e6" (UID: "afe14932-62ea-4859-95eb-f030280ac1e6"). InnerVolumeSpecName "kube-api-access-zr7v2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:04:22.248190 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:22.247873    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zr7v2\" (UniqueName: \"kubernetes.io/projected/afe14932-62ea-4859-95eb-f030280ac1e6-kube-api-access-zr7v2\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:04:22.325768 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:22.325688    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:22.867383 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:22.867338    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:22.985750 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:22.985651    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:22.994869 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:22.994833    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b0fba292-65679f777d-tkn74"
Oct 30 05:04:23.077931 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:23.073522    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b0fba292-65679f777d-tkn74"]
Oct 30 05:04:23.214979 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:23.211957    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b0fba292-65679f777d-tkn74"]
Oct 30 05:04:23.665321 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:23.665286    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-0b21ba/distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6"
Oct 30 05:04:23.682672 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:23.682647    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-0b21ba/distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6"
Oct 30 05:04:24.033839 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:24.033229    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0b21ba/distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6"
Oct 30 05:04:24.660187 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:24.660151    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:24.747585 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:24.747551    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:24.872928 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:24.865510    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:24.913263 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:24.913167    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afe14932-62ea-4859-95eb-f030280ac1e6" path="/var/lib/kubelet/pods/afe14932-62ea-4859-95eb-f030280ac1e6/volumes"
Oct 30 05:04:25.079764 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:25.079717    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:04:25.080102 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:25.079800    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:04:26.032468 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:26.032427    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ft7s9" event={"ID":"f7b35ef1-25f4-46ac-9a62-fe5e2e027af4","Type":"ContainerStarted","Data":"a5371caf43a0c6bd2a4fc08db40fc7403987f1819121eaeae3ad981bfa6cb664"}
Oct 30 05:04:26.032652 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:26.032493    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ft7s9"
Oct 30 05:04:26.034561 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:26.034533    5898 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-ft7s9 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 05:04:26.034689 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:26.034586    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ft7s9" podUID="f7b35ef1-25f4-46ac-9a62-fe5e2e027af4" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 05:04:26.090022 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:26.089968    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ft7s9" podStartSLOduration=1.962176335 podStartE2EDuration="7.089954013s" totalImagesPullingTime="5.127777678s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:04:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:04:26.085985102 +0000 UTC m=+1195.309805028" watchObservedRunningTime="2026-10-30 05:04:26.089954013 +0000 UTC m=+1195.313773914"
Oct 30 05:04:26.447170 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:26.444808    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-l8bcz"]
Oct 30 05:04:26.447170 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:26.445717    5898 cpu_manager.go:414] "containerMap: removing container" podUID="afe14932-62ea-4859-95eb-f030280ac1e6" containerName="payload-processing"
Oct 30 05:04:26.447170 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:26.445734    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afe14932-62ea-4859-95eb-f030280ac1e6" containerName="payload-processing"
Oct 30 05:04:26.447170 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:26.445929    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="afe14932-62ea-4859-95eb-f030280ac1e6" containerName="payload-processing"
Oct 30 05:04:26.450330 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:26.450309    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-l8bcz"
Oct 30 05:04:26.466828 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:26.466800    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-l8bcz"]
Oct 30 05:04:26.471702 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:26.471682    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-b0fba292-openshift-default-dockercfg-ql9zz\""
Oct 30 05:04:26.576203 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:26.576170    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/860aa2af-bde4-41f8-aa0c-60d3ccaf97a2-istio-token\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-l8bcz\" (UID: \"860aa2af-bde4-41f8-aa0c-60d3ccaf97a2\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-l8bcz"
Oct 30 05:04:26.576380 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:26.576209    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/860aa2af-bde4-41f8-aa0c-60d3ccaf97a2-workload-certs\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-l8bcz\" (UID: \"860aa2af-bde4-41f8-aa0c-60d3ccaf97a2\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-l8bcz"
Oct 30 05:04:26.576380 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:26.576326    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/860aa2af-bde4-41f8-aa0c-60d3ccaf97a2-istio-envoy\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-l8bcz\" (UID: \"860aa2af-bde4-41f8-aa0c-60d3ccaf97a2\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-l8bcz"
Oct 30 05:04:26.576380 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:26.576349    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/860aa2af-bde4-41f8-aa0c-60d3ccaf97a2-istiod-ca-cert\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-l8bcz\" (UID: \"860aa2af-bde4-41f8-aa0c-60d3ccaf97a2\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-l8bcz"
Oct 30 05:04:26.576542 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:26.576492    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/860aa2af-bde4-41f8-aa0c-60d3ccaf97a2-credential-socket\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-l8bcz\" (UID: \"860aa2af-bde4-41f8-aa0c-60d3ccaf97a2\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-l8bcz"
Oct 30 05:04:26.576620 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:26.576605    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv6m9\" (UniqueName: \"kubernetes.io/projected/860aa2af-bde4-41f8-aa0c-60d3ccaf97a2-kube-api-access-bv6m9\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-l8bcz\" (UID: \"860aa2af-bde4-41f8-aa0c-60d3ccaf97a2\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-l8bcz"
Oct 30 05:04:26.576674 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:26.576635    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/860aa2af-bde4-41f8-aa0c-60d3ccaf97a2-istio-podinfo\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-l8bcz\" (UID: \"860aa2af-bde4-41f8-aa0c-60d3ccaf97a2\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-l8bcz"
Oct 30 05:04:26.576757 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:26.576738    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/860aa2af-bde4-41f8-aa0c-60d3ccaf97a2-workload-socket\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-l8bcz\" (UID: \"860aa2af-bde4-41f8-aa0c-60d3ccaf97a2\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-l8bcz"
Oct 30 05:04:26.576815 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:26.576790    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/860aa2af-bde4-41f8-aa0c-60d3ccaf97a2-istio-data\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-l8bcz\" (UID: \"860aa2af-bde4-41f8-aa0c-60d3ccaf97a2\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-l8bcz"
Oct 30 05:04:26.678261 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:26.678228    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/860aa2af-bde4-41f8-aa0c-60d3ccaf97a2-istio-data\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-l8bcz\" (UID: \"860aa2af-bde4-41f8-aa0c-60d3ccaf97a2\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-l8bcz"
Oct 30 05:04:26.678422 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:26.678305    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/860aa2af-bde4-41f8-aa0c-60d3ccaf97a2-istio-token\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-l8bcz\" (UID: \"860aa2af-bde4-41f8-aa0c-60d3ccaf97a2\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-l8bcz"
Oct 30 05:04:26.678422 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:26.678327    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/860aa2af-bde4-41f8-aa0c-60d3ccaf97a2-workload-certs\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-l8bcz\" (UID: \"860aa2af-bde4-41f8-aa0c-60d3ccaf97a2\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-l8bcz"
Oct 30 05:04:26.678508 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:26.678485    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/860aa2af-bde4-41f8-aa0c-60d3ccaf97a2-istio-envoy\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-l8bcz\" (UID: \"860aa2af-bde4-41f8-aa0c-60d3ccaf97a2\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-l8bcz"
Oct 30 05:04:26.678547 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:26.678534    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/860aa2af-bde4-41f8-aa0c-60d3ccaf97a2-istiod-ca-cert\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-l8bcz\" (UID: \"860aa2af-bde4-41f8-aa0c-60d3ccaf97a2\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-l8bcz"
Oct 30 05:04:26.678658 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:26.678641    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/860aa2af-bde4-41f8-aa0c-60d3ccaf97a2-credential-socket\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-l8bcz\" (UID: \"860aa2af-bde4-41f8-aa0c-60d3ccaf97a2\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-l8bcz"
Oct 30 05:04:26.678658 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:26.678649    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/860aa2af-bde4-41f8-aa0c-60d3ccaf97a2-istio-data\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-l8bcz\" (UID: \"860aa2af-bde4-41f8-aa0c-60d3ccaf97a2\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-l8bcz"
Oct 30 05:04:26.678779 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:26.678671    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/860aa2af-bde4-41f8-aa0c-60d3ccaf97a2-workload-certs\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-l8bcz\" (UID: \"860aa2af-bde4-41f8-aa0c-60d3ccaf97a2\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-l8bcz"
Oct 30 05:04:26.678779 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:26.678764    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv6m9\" (UniqueName: \"kubernetes.io/projected/860aa2af-bde4-41f8-aa0c-60d3ccaf97a2-kube-api-access-bv6m9\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-l8bcz\" (UID: \"860aa2af-bde4-41f8-aa0c-60d3ccaf97a2\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-l8bcz"
Oct 30 05:04:26.678887 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:26.678801    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/860aa2af-bde4-41f8-aa0c-60d3ccaf97a2-istio-podinfo\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-l8bcz\" (UID: \"860aa2af-bde4-41f8-aa0c-60d3ccaf97a2\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-l8bcz"
Oct 30 05:04:26.679011 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:26.678916    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/860aa2af-bde4-41f8-aa0c-60d3ccaf97a2-workload-socket\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-l8bcz\" (UID: \"860aa2af-bde4-41f8-aa0c-60d3ccaf97a2\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-l8bcz"
Oct 30 05:04:26.679011 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:26.678895    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/860aa2af-bde4-41f8-aa0c-60d3ccaf97a2-credential-socket\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-l8bcz\" (UID: \"860aa2af-bde4-41f8-aa0c-60d3ccaf97a2\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-l8bcz"
Oct 30 05:04:26.679251 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:26.679232    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/860aa2af-bde4-41f8-aa0c-60d3ccaf97a2-workload-socket\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-l8bcz\" (UID: \"860aa2af-bde4-41f8-aa0c-60d3ccaf97a2\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-l8bcz"
Oct 30 05:04:26.679338 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:26.679317    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/860aa2af-bde4-41f8-aa0c-60d3ccaf97a2-istiod-ca-cert\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-l8bcz\" (UID: \"860aa2af-bde4-41f8-aa0c-60d3ccaf97a2\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-l8bcz"
Oct 30 05:04:26.680824 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:26.680806    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/860aa2af-bde4-41f8-aa0c-60d3ccaf97a2-istio-envoy\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-l8bcz\" (UID: \"860aa2af-bde4-41f8-aa0c-60d3ccaf97a2\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-l8bcz"
Oct 30 05:04:26.681274 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:26.681254    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/860aa2af-bde4-41f8-aa0c-60d3ccaf97a2-istio-podinfo\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-l8bcz\" (UID: \"860aa2af-bde4-41f8-aa0c-60d3ccaf97a2\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-l8bcz"
Oct 30 05:04:26.696671 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:26.696648    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/860aa2af-bde4-41f8-aa0c-60d3ccaf97a2-istio-token\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-l8bcz\" (UID: \"860aa2af-bde4-41f8-aa0c-60d3ccaf97a2\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-l8bcz"
Oct 30 05:04:26.697075 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:26.697053    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv6m9\" (UniqueName: \"kubernetes.io/projected/860aa2af-bde4-41f8-aa0c-60d3ccaf97a2-kube-api-access-bv6m9\") pod \"e2e-ait-b0fba292-openshift-default-658566db69-l8bcz\" (UID: \"860aa2af-bde4-41f8-aa0c-60d3ccaf97a2\") " pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-l8bcz"
Oct 30 05:04:26.768559 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:26.768495    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-l8bcz"
Oct 30 05:04:26.828380 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:04:26.827954    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:04:26.933950 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:26.933921    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:27.042334 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:27.042247    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ft7s9"
Oct 30 05:04:27.095776 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:27.095744    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-l8bcz"]
Oct 30 05:04:27.104964 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:27.104928    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:04:27.105079 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:27.105010    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:04:27.105079 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:27.105056    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:04:28.050614 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:28.050563    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-l8bcz" event={"ID":"860aa2af-bde4-41f8-aa0c-60d3ccaf97a2","Type":"ContainerStarted","Data":"339959782dd108a5b7f013f521dd91dd17896ed91b3dd88300dfb7e6a2fae9a6"}
Oct 30 05:04:28.050824 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:28.050671    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-l8bcz" event={"ID":"860aa2af-bde4-41f8-aa0c-60d3ccaf97a2","Type":"ContainerStarted","Data":"0fb24791d075b9dc2cb3a92c74ec5b5c8ffa7b2faecf37cd105a35801427e77d"}
Oct 30 05:04:28.132852 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:28.132802    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-l8bcz" podStartSLOduration=2.132787294 podStartE2EDuration="2.132787294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:04:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:04:28.11815804 +0000 UTC m=+1197.341977940" watchObservedRunningTime="2026-10-30 05:04:28.132787294 +0000 UTC m=+1197.356607195"
Oct 30 05:04:28.515741 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:28.515697    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b0fba292-c7b9d9646-l4fz5"]
Oct 30 05:04:28.515999 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:28.515951    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0fba292-c7b9d9646-l4fz5" podUID="2163caad-3db0-4f6f-9e99-f3c0a83cb50e" containerName="maas-api" containerID="cri-o://56efb6814892c5e8b5c628cc197adbe38fa531fa26529f6d809fb5ddde8108b9" gracePeriod=30
Oct 30 05:04:28.607488 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:28.607440    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:28.776979 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:28.769316    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-l8bcz"
Oct 30 05:04:28.776979 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:28.772534    5898 patch_prober.go:29] interesting pod/e2e-ait-b0fba292-openshift-default-658566db69-l8bcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 05:04:28.776979 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:28.772590    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-l8bcz" podUID="860aa2af-bde4-41f8-aa0c-60d3ccaf97a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 05:04:29.046841 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:29.046769    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0fba292-c7b9d9646-l4fz5"
Oct 30 05:04:29.061754 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:29.061711    5898 generic.go:356] "Generic (PLEG): container finished" podID="2163caad-3db0-4f6f-9e99-f3c0a83cb50e" containerID="56efb6814892c5e8b5c628cc197adbe38fa531fa26529f6d809fb5ddde8108b9" exitCode=0
Oct 30 05:04:29.061933 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:29.061756    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0fba292-c7b9d9646-l4fz5"
Oct 30 05:04:29.061933 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:29.061818    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0fba292-c7b9d9646-l4fz5" event={"ID":"2163caad-3db0-4f6f-9e99-f3c0a83cb50e","Type":"ContainerDied","Data":"56efb6814892c5e8b5c628cc197adbe38fa531fa26529f6d809fb5ddde8108b9"}
Oct 30 05:04:29.061933 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:29.061854    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0fba292-c7b9d9646-l4fz5" event={"ID":"2163caad-3db0-4f6f-9e99-f3c0a83cb50e","Type":"ContainerDied","Data":"25bcd98972330a384f4dd62e5e1df264005dd12285963cf8755e110fea55d894"}
Oct 30 05:04:29.061933 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:29.061887    5898 scope.go:122] "RemoveContainer" containerID="56efb6814892c5e8b5c628cc197adbe38fa531fa26529f6d809fb5ddde8108b9"
Oct 30 05:04:29.080107 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:29.078118    5898 scope.go:122] "RemoveContainer" containerID="56efb6814892c5e8b5c628cc197adbe38fa531fa26529f6d809fb5ddde8108b9"
Oct 30 05:04:29.080107 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:04:29.079732    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56efb6814892c5e8b5c628cc197adbe38fa531fa26529f6d809fb5ddde8108b9\": container with ID starting with 56efb6814892c5e8b5c628cc197adbe38fa531fa26529f6d809fb5ddde8108b9 not found: ID does not exist" containerID="56efb6814892c5e8b5c628cc197adbe38fa531fa26529f6d809fb5ddde8108b9"
Oct 30 05:04:29.080107 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:29.079771    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56efb6814892c5e8b5c628cc197adbe38fa531fa26529f6d809fb5ddde8108b9"} err="failed to get container status \"56efb6814892c5e8b5c628cc197adbe38fa531fa26529f6d809fb5ddde8108b9\": rpc error: code = NotFound desc = could not find container \"56efb6814892c5e8b5c628cc197adbe38fa531fa26529f6d809fb5ddde8108b9\": container with ID starting with 56efb6814892c5e8b5c628cc197adbe38fa531fa26529f6d809fb5ddde8108b9 not found: ID does not exist"
Oct 30 05:04:29.243822 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:29.243785    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2163caad-3db0-4f6f-9e99-f3c0a83cb50e-kube-api-access-f2qkr\" (UniqueName: \"kubernetes.io/projected/2163caad-3db0-4f6f-9e99-f3c0a83cb50e-kube-api-access-f2qkr\") pod \"2163caad-3db0-4f6f-9e99-f3c0a83cb50e\" (UID: \"2163caad-3db0-4f6f-9e99-f3c0a83cb50e\") "
Oct 30 05:04:29.244405 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:29.244371    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2163caad-3db0-4f6f-9e99-f3c0a83cb50e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2163caad-3db0-4f6f-9e99-f3c0a83cb50e-maas-api-tls\") pod \"2163caad-3db0-4f6f-9e99-f3c0a83cb50e\" (UID: \"2163caad-3db0-4f6f-9e99-f3c0a83cb50e\") "
Oct 30 05:04:29.244630 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:29.244613    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2163caad-3db0-4f6f-9e99-f3c0a83cb50e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2163caad-3db0-4f6f-9e99-f3c0a83cb50e-openshift-service-ca\") pod \"2163caad-3db0-4f6f-9e99-f3c0a83cb50e\" (UID: \"2163caad-3db0-4f6f-9e99-f3c0a83cb50e\") "
Oct 30 05:04:29.244989 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:29.244968    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2163caad-3db0-4f6f-9e99-f3c0a83cb50e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2163caad-3db0-4f6f-9e99-f3c0a83cb50e-metrics-certs\") pod \"2163caad-3db0-4f6f-9e99-f3c0a83cb50e\" (UID: \"2163caad-3db0-4f6f-9e99-f3c0a83cb50e\") "
Oct 30 05:04:29.260446 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:29.260419    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2163caad-3db0-4f6f-9e99-f3c0a83cb50e-openshift-service-ca" pod "2163caad-3db0-4f6f-9e99-f3c0a83cb50e" (UID: "2163caad-3db0-4f6f-9e99-f3c0a83cb50e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:04:29.270460 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:29.270428    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:29.272766 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:29.272724    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2163caad-3db0-4f6f-9e99-f3c0a83cb50e-maas-api-tls" pod "2163caad-3db0-4f6f-9e99-f3c0a83cb50e" (UID: "2163caad-3db0-4f6f-9e99-f3c0a83cb50e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:04:29.319978 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:29.301074    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2163caad-3db0-4f6f-9e99-f3c0a83cb50e-kube-api-access-f2qkr" pod "2163caad-3db0-4f6f-9e99-f3c0a83cb50e" (UID: "2163caad-3db0-4f6f-9e99-f3c0a83cb50e"). InnerVolumeSpecName "kube-api-access-f2qkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:04:29.336160 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:29.336108    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2163caad-3db0-4f6f-9e99-f3c0a83cb50e-metrics-certs" pod "2163caad-3db0-4f6f-9e99-f3c0a83cb50e" (UID: "2163caad-3db0-4f6f-9e99-f3c0a83cb50e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:04:29.361239 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:29.348460    5898 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2163caad-3db0-4f6f-9e99-f3c0a83cb50e-openshift-service-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:04:29.361239 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:29.348497    5898 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2163caad-3db0-4f6f-9e99-f3c0a83cb50e-metrics-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:04:29.361239 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:29.348512    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2qkr\" (UniqueName: \"kubernetes.io/projected/2163caad-3db0-4f6f-9e99-f3c0a83cb50e-kube-api-access-f2qkr\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:04:29.361239 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:29.348525    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2163caad-3db0-4f6f-9e99-f3c0a83cb50e-maas-api-tls\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:04:29.417609 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:04:29.417575    5898 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2163caad_3db0_4f6f_9e99_f3c0a83cb50e.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 05:04:29.430258 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:29.430219    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:29.436878 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:29.436849    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b0fba292-c7b9d9646-l4fz5"]
Oct 30 05:04:29.449299 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:29.449274    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b0fba292-c7b9d9646-l4fz5"]
Oct 30 05:04:29.623270 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:29.623226    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:29.769831 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:29.769800    5898 patch_prober.go:29] interesting pod/e2e-ait-b0fba292-openshift-default-658566db69-l8bcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 05:04:29.770014 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:29.769857    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-l8bcz" podUID="860aa2af-bde4-41f8-aa0c-60d3ccaf97a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 05:04:30.769634 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:30.769600    5898 patch_prober.go:29] interesting pod/e2e-ait-b0fba292-openshift-default-658566db69-l8bcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 05:04:30.770015 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:30.769666    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-l8bcz" podUID="860aa2af-bde4-41f8-aa0c-60d3ccaf97a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 05:04:30.908786 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:30.908753    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2163caad-3db0-4f6f-9e99-f3c0a83cb50e" path="/var/lib/kubelet/pods/2163caad-3db0-4f6f-9e99-f3c0a83cb50e/volumes"
Oct 30 05:04:31.147973 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:31.147928    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-l8bcz"]
Oct 30 05:04:31.148194 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:31.148166    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-l8bcz" podUID="860aa2af-bde4-41f8-aa0c-60d3ccaf97a2" containerName="istio-proxy" containerID="cri-o://339959782dd108a5b7f013f521dd91dd17896ed91b3dd88300dfb7e6a2fae9a6" gracePeriod=30
Oct 30 05:04:31.158776 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:31.158752    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:31.786980 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:31.783011    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:32.112570 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:32.112527    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:32.266212 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:32.266169    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:32.541374 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:32.541257    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:33.971186 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:33.971147    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:35.618557 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:35.618517    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:35.736753 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:35.736722    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:36.327606 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:36.327533    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:37.283087 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:37.283055    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:37.381586 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:37.381545    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:38.253506 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:38.253470    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:38.333026 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:38.332986    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd"]
Oct 30 05:04:38.334543 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:38.334513    5898 cpu_manager.go:414] "containerMap: removing container" podUID="2163caad-3db0-4f6f-9e99-f3c0a83cb50e" containerName="maas-api"
Oct 30 05:04:38.334543 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:38.334542    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2163caad-3db0-4f6f-9e99-f3c0a83cb50e" containerName="maas-api"
Oct 30 05:04:38.334757 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:38.334698    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="2163caad-3db0-4f6f-9e99-f3c0a83cb50e" containerName="maas-api"
Oct 30 05:04:38.341112 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:38.341083    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd"
Oct 30 05:04:38.354869 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:38.354841    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd"]
Oct 30 05:04:38.460252 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:38.460216    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/736153cf-ff1d-4f3e-b460-a62cd005c988-workload-socket\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd\" (UID: \"736153cf-ff1d-4f3e-b460-a62cd005c988\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd"
Oct 30 05:04:38.460500 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:38.460273    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/736153cf-ff1d-4f3e-b460-a62cd005c988-istiod-ca-cert\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd\" (UID: \"736153cf-ff1d-4f3e-b460-a62cd005c988\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd"
Oct 30 05:04:38.460500 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:38.460388    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/736153cf-ff1d-4f3e-b460-a62cd005c988-workload-certs\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd\" (UID: \"736153cf-ff1d-4f3e-b460-a62cd005c988\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd"
Oct 30 05:04:38.460500 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:38.460486    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/736153cf-ff1d-4f3e-b460-a62cd005c988-istio-token\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd\" (UID: \"736153cf-ff1d-4f3e-b460-a62cd005c988\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd"
Oct 30 05:04:38.460702 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:38.460556    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/736153cf-ff1d-4f3e-b460-a62cd005c988-istio-data\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd\" (UID: \"736153cf-ff1d-4f3e-b460-a62cd005c988\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd"
Oct 30 05:04:38.460702 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:38.460591    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-679r9\" (UniqueName: \"kubernetes.io/projected/736153cf-ff1d-4f3e-b460-a62cd005c988-kube-api-access-679r9\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd\" (UID: \"736153cf-ff1d-4f3e-b460-a62cd005c988\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd"
Oct 30 05:04:38.460702 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:38.460677    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/736153cf-ff1d-4f3e-b460-a62cd005c988-credential-socket\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd\" (UID: \"736153cf-ff1d-4f3e-b460-a62cd005c988\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd"
Oct 30 05:04:38.460822 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:38.460712    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/736153cf-ff1d-4f3e-b460-a62cd005c988-istio-envoy\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd\" (UID: \"736153cf-ff1d-4f3e-b460-a62cd005c988\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd"
Oct 30 05:04:38.460822 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:38.460745    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/736153cf-ff1d-4f3e-b460-a62cd005c988-istio-podinfo\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd\" (UID: \"736153cf-ff1d-4f3e-b460-a62cd005c988\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd"
Oct 30 05:04:38.562249 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:38.562160    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/736153cf-ff1d-4f3e-b460-a62cd005c988-istio-data\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd\" (UID: \"736153cf-ff1d-4f3e-b460-a62cd005c988\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd"
Oct 30 05:04:38.562249 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:38.562225    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-679r9\" (UniqueName: \"kubernetes.io/projected/736153cf-ff1d-4f3e-b460-a62cd005c988-kube-api-access-679r9\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd\" (UID: \"736153cf-ff1d-4f3e-b460-a62cd005c988\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd"
Oct 30 05:04:38.562472 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:38.562259    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/736153cf-ff1d-4f3e-b460-a62cd005c988-credential-socket\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd\" (UID: \"736153cf-ff1d-4f3e-b460-a62cd005c988\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd"
Oct 30 05:04:38.562472 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:38.562301    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/736153cf-ff1d-4f3e-b460-a62cd005c988-istio-envoy\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd\" (UID: \"736153cf-ff1d-4f3e-b460-a62cd005c988\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd"
Oct 30 05:04:38.562472 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:38.562351    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/736153cf-ff1d-4f3e-b460-a62cd005c988-istio-podinfo\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd\" (UID: \"736153cf-ff1d-4f3e-b460-a62cd005c988\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd"
Oct 30 05:04:38.562630 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:38.562548    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/736153cf-ff1d-4f3e-b460-a62cd005c988-workload-socket\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd\" (UID: \"736153cf-ff1d-4f3e-b460-a62cd005c988\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd"
Oct 30 05:04:38.562630 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:38.562605    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/736153cf-ff1d-4f3e-b460-a62cd005c988-istiod-ca-cert\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd\" (UID: \"736153cf-ff1d-4f3e-b460-a62cd005c988\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd"
Oct 30 05:04:38.562630 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:38.562615    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/736153cf-ff1d-4f3e-b460-a62cd005c988-istio-data\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd\" (UID: \"736153cf-ff1d-4f3e-b460-a62cd005c988\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd"
Oct 30 05:04:38.562782 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:38.562713    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/736153cf-ff1d-4f3e-b460-a62cd005c988-credential-socket\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd\" (UID: \"736153cf-ff1d-4f3e-b460-a62cd005c988\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd"
Oct 30 05:04:38.562782 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:38.562745    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/736153cf-ff1d-4f3e-b460-a62cd005c988-workload-certs\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd\" (UID: \"736153cf-ff1d-4f3e-b460-a62cd005c988\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd"
Oct 30 05:04:38.562894 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:38.562827    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/736153cf-ff1d-4f3e-b460-a62cd005c988-istio-token\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd\" (UID: \"736153cf-ff1d-4f3e-b460-a62cd005c988\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd"
Oct 30 05:04:38.562894 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:38.562863    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/736153cf-ff1d-4f3e-b460-a62cd005c988-workload-socket\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd\" (UID: \"736153cf-ff1d-4f3e-b460-a62cd005c988\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd"
Oct 30 05:04:38.563126 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:38.563102    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/736153cf-ff1d-4f3e-b460-a62cd005c988-workload-certs\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd\" (UID: \"736153cf-ff1d-4f3e-b460-a62cd005c988\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd"
Oct 30 05:04:38.563327 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:38.563309    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/736153cf-ff1d-4f3e-b460-a62cd005c988-istiod-ca-cert\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd\" (UID: \"736153cf-ff1d-4f3e-b460-a62cd005c988\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd"
Oct 30 05:04:38.564926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:38.564887    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/736153cf-ff1d-4f3e-b460-a62cd005c988-istio-envoy\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd\" (UID: \"736153cf-ff1d-4f3e-b460-a62cd005c988\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd"
Oct 30 05:04:38.565044 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:38.564970    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/736153cf-ff1d-4f3e-b460-a62cd005c988-istio-podinfo\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd\" (UID: \"736153cf-ff1d-4f3e-b460-a62cd005c988\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd"
Oct 30 05:04:38.581051 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:38.581027    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-679r9\" (UniqueName: \"kubernetes.io/projected/736153cf-ff1d-4f3e-b460-a62cd005c988-kube-api-access-679r9\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd\" (UID: \"736153cf-ff1d-4f3e-b460-a62cd005c988\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd"
Oct 30 05:04:38.581138 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:38.581115    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/736153cf-ff1d-4f3e-b460-a62cd005c988-istio-token\") pod \"e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd\" (UID: \"736153cf-ff1d-4f3e-b460-a62cd005c988\") " pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd"
Oct 30 05:04:38.659465 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:38.659436    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd"
Oct 30 05:04:38.929334 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:38.929305    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-l8bcz"
Oct 30 05:04:38.981659 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:04:38.981622    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod736153cf_ff1d_4f3e_b460_a62cd005c988.slice/crio-1a90e73417e86428876242c0fb075c3d94b9598bd05771933a826d6d31816b4e WatchSource:0}: Error finding container 1a90e73417e86428876242c0fb075c3d94b9598bd05771933a826d6d31816b4e: Status 404 returned error can't find the container with id 1a90e73417e86428876242c0fb075c3d94b9598bd05771933a826d6d31816b4e
Oct 30 05:04:38.987671 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:38.987623    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:04:38.987813 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:38.987723    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:04:38.987813 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:38.987769    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:04:38.991202 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:38.991172    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd"]
Oct 30 05:04:39.071381 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:39.071354    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/860aa2af-bde4-41f8-aa0c-60d3ccaf97a2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/860aa2af-bde4-41f8-aa0c-60d3ccaf97a2-istio-envoy\") pod \"860aa2af-bde4-41f8-aa0c-60d3ccaf97a2\" (UID: \"860aa2af-bde4-41f8-aa0c-60d3ccaf97a2\") "
Oct 30 05:04:39.071517 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:39.071501    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/860aa2af-bde4-41f8-aa0c-60d3ccaf97a2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/860aa2af-bde4-41f8-aa0c-60d3ccaf97a2-istiod-ca-cert\") pod \"860aa2af-bde4-41f8-aa0c-60d3ccaf97a2\" (UID: \"860aa2af-bde4-41f8-aa0c-60d3ccaf97a2\") "
Oct 30 05:04:39.071563 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:39.071543    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/860aa2af-bde4-41f8-aa0c-60d3ccaf97a2-kube-api-access-bv6m9\" (UniqueName: \"kubernetes.io/projected/860aa2af-bde4-41f8-aa0c-60d3ccaf97a2-kube-api-access-bv6m9\") pod \"860aa2af-bde4-41f8-aa0c-60d3ccaf97a2\" (UID: \"860aa2af-bde4-41f8-aa0c-60d3ccaf97a2\") "
Oct 30 05:04:39.071665 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:39.071644    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/860aa2af-bde4-41f8-aa0c-60d3ccaf97a2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/860aa2af-bde4-41f8-aa0c-60d3ccaf97a2-credential-socket\") pod \"860aa2af-bde4-41f8-aa0c-60d3ccaf97a2\" (UID: \"860aa2af-bde4-41f8-aa0c-60d3ccaf97a2\") "
Oct 30 05:04:39.071817 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:39.071725    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/860aa2af-bde4-41f8-aa0c-60d3ccaf97a2-istio-token\" (UniqueName: \"kubernetes.io/projected/860aa2af-bde4-41f8-aa0c-60d3ccaf97a2-istio-token\") pod \"860aa2af-bde4-41f8-aa0c-60d3ccaf97a2\" (UID: \"860aa2af-bde4-41f8-aa0c-60d3ccaf97a2\") "
Oct 30 05:04:39.071817 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:39.071777    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/860aa2af-bde4-41f8-aa0c-60d3ccaf97a2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/860aa2af-bde4-41f8-aa0c-60d3ccaf97a2-workload-certs\") pod \"860aa2af-bde4-41f8-aa0c-60d3ccaf97a2\" (UID: \"860aa2af-bde4-41f8-aa0c-60d3ccaf97a2\") "
Oct 30 05:04:39.071969 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:39.071827    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/860aa2af-bde4-41f8-aa0c-60d3ccaf97a2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/860aa2af-bde4-41f8-aa0c-60d3ccaf97a2-istio-data\") pod \"860aa2af-bde4-41f8-aa0c-60d3ccaf97a2\" (UID: \"860aa2af-bde4-41f8-aa0c-60d3ccaf97a2\") "
Oct 30 05:04:39.071969 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:39.071931    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/860aa2af-bde4-41f8-aa0c-60d3ccaf97a2-istiod-ca-cert" pod "860aa2af-bde4-41f8-aa0c-60d3ccaf97a2" (UID: "860aa2af-bde4-41f8-aa0c-60d3ccaf97a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:04:39.071969 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:39.071947    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/860aa2af-bde4-41f8-aa0c-60d3ccaf97a2-credential-socket" pod "860aa2af-bde4-41f8-aa0c-60d3ccaf97a2" (UID: "860aa2af-bde4-41f8-aa0c-60d3ccaf97a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:04:39.072128 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:39.071970    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/860aa2af-bde4-41f8-aa0c-60d3ccaf97a2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/860aa2af-bde4-41f8-aa0c-60d3ccaf97a2-istio-podinfo\") pod \"860aa2af-bde4-41f8-aa0c-60d3ccaf97a2\" (UID: \"860aa2af-bde4-41f8-aa0c-60d3ccaf97a2\") "
Oct 30 05:04:39.072128 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:39.072018    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/860aa2af-bde4-41f8-aa0c-60d3ccaf97a2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/860aa2af-bde4-41f8-aa0c-60d3ccaf97a2-workload-socket\") pod \"860aa2af-bde4-41f8-aa0c-60d3ccaf97a2\" (UID: \"860aa2af-bde4-41f8-aa0c-60d3ccaf97a2\") "
Oct 30 05:04:39.072239 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:39.072172    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/860aa2af-bde4-41f8-aa0c-60d3ccaf97a2-workload-certs" pod "860aa2af-bde4-41f8-aa0c-60d3ccaf97a2" (UID: "860aa2af-bde4-41f8-aa0c-60d3ccaf97a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:04:39.072366 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:39.072347    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/860aa2af-bde4-41f8-aa0c-60d3ccaf97a2-workload-socket" pod "860aa2af-bde4-41f8-aa0c-60d3ccaf97a2" (UID: "860aa2af-bde4-41f8-aa0c-60d3ccaf97a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:04:39.072476 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:39.072455    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/860aa2af-bde4-41f8-aa0c-60d3ccaf97a2-istio-data" pod "860aa2af-bde4-41f8-aa0c-60d3ccaf97a2" (UID: "860aa2af-bde4-41f8-aa0c-60d3ccaf97a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:04:39.072799 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:39.072782    5898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/860aa2af-bde4-41f8-aa0c-60d3ccaf97a2-istiod-ca-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:04:39.072888 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:39.072804    5898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/860aa2af-bde4-41f8-aa0c-60d3ccaf97a2-credential-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:04:39.072888 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:39.072822    5898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/860aa2af-bde4-41f8-aa0c-60d3ccaf97a2-workload-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:04:39.072888 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:39.072836    5898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/860aa2af-bde4-41f8-aa0c-60d3ccaf97a2-istio-data\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:04:39.072888 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:39.072848    5898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/860aa2af-bde4-41f8-aa0c-60d3ccaf97a2-workload-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:04:39.073461 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:39.073436    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/860aa2af-bde4-41f8-aa0c-60d3ccaf97a2-istio-envoy" pod "860aa2af-bde4-41f8-aa0c-60d3ccaf97a2" (UID: "860aa2af-bde4-41f8-aa0c-60d3ccaf97a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:04:39.074039 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:39.074013    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/860aa2af-bde4-41f8-aa0c-60d3ccaf97a2-istio-podinfo" pod "860aa2af-bde4-41f8-aa0c-60d3ccaf97a2" (UID: "860aa2af-bde4-41f8-aa0c-60d3ccaf97a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:04:39.074299 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:39.074278    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/860aa2af-bde4-41f8-aa0c-60d3ccaf97a2-kube-api-access-bv6m9" pod "860aa2af-bde4-41f8-aa0c-60d3ccaf97a2" (UID: "860aa2af-bde4-41f8-aa0c-60d3ccaf97a2"). InnerVolumeSpecName "kube-api-access-bv6m9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:04:39.074526 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:39.074506    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/860aa2af-bde4-41f8-aa0c-60d3ccaf97a2-istio-token" pod "860aa2af-bde4-41f8-aa0c-60d3ccaf97a2" (UID: "860aa2af-bde4-41f8-aa0c-60d3ccaf97a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:04:39.174250 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:39.174173    5898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/860aa2af-bde4-41f8-aa0c-60d3ccaf97a2-istio-envoy\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:04:39.174250 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:39.174204    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bv6m9\" (UniqueName: \"kubernetes.io/projected/860aa2af-bde4-41f8-aa0c-60d3ccaf97a2-kube-api-access-bv6m9\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:04:39.174250 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:39.174214    5898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/860aa2af-bde4-41f8-aa0c-60d3ccaf97a2-istio-token\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:04:39.174250 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:39.174228    5898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/860aa2af-bde4-41f8-aa0c-60d3ccaf97a2-istio-podinfo\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:04:39.201784 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:39.201751    5898 generic.go:356] "Generic (PLEG): container finished" podID="860aa2af-bde4-41f8-aa0c-60d3ccaf97a2" containerID="339959782dd108a5b7f013f521dd91dd17896ed91b3dd88300dfb7e6a2fae9a6" exitCode=0
Oct 30 05:04:39.201990 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:39.201822    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-l8bcz"
Oct 30 05:04:39.201990 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:39.201850    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-l8bcz" event={"ID":"860aa2af-bde4-41f8-aa0c-60d3ccaf97a2","Type":"ContainerDied","Data":"339959782dd108a5b7f013f521dd91dd17896ed91b3dd88300dfb7e6a2fae9a6"}
Oct 30 05:04:39.201990 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:39.201978    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-l8bcz" event={"ID":"860aa2af-bde4-41f8-aa0c-60d3ccaf97a2","Type":"ContainerDied","Data":"0fb24791d075b9dc2cb3a92c74ec5b5c8ffa7b2faecf37cd105a35801427e77d"}
Oct 30 05:04:39.202161 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:39.202047    5898 scope.go:122] "RemoveContainer" containerID="339959782dd108a5b7f013f521dd91dd17896ed91b3dd88300dfb7e6a2fae9a6"
Oct 30 05:04:39.210988 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:39.210960    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd" event={"ID":"736153cf-ff1d-4f3e-b460-a62cd005c988","Type":"ContainerStarted","Data":"7b778252e921f65e90058efdae9bb9f5e44b78bdff216dda41722e2e5957c15d"}
Oct 30 05:04:39.211120 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:39.211020    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd" event={"ID":"736153cf-ff1d-4f3e-b460-a62cd005c988","Type":"ContainerStarted","Data":"1a90e73417e86428876242c0fb075c3d94b9598bd05771933a826d6d31816b4e"}
Oct 30 05:04:39.214801 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:39.214782    5898 scope.go:122] "RemoveContainer" containerID="339959782dd108a5b7f013f521dd91dd17896ed91b3dd88300dfb7e6a2fae9a6"
Oct 30 05:04:39.215213 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:04:39.215126    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"339959782dd108a5b7f013f521dd91dd17896ed91b3dd88300dfb7e6a2fae9a6\": container with ID starting with 339959782dd108a5b7f013f521dd91dd17896ed91b3dd88300dfb7e6a2fae9a6 not found: ID does not exist" containerID="339959782dd108a5b7f013f521dd91dd17896ed91b3dd88300dfb7e6a2fae9a6"
Oct 30 05:04:39.215315 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:39.215210    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"339959782dd108a5b7f013f521dd91dd17896ed91b3dd88300dfb7e6a2fae9a6"} err="failed to get container status \"339959782dd108a5b7f013f521dd91dd17896ed91b3dd88300dfb7e6a2fae9a6\": rpc error: code = NotFound desc = could not find container \"339959782dd108a5b7f013f521dd91dd17896ed91b3dd88300dfb7e6a2fae9a6\": container with ID starting with 339959782dd108a5b7f013f521dd91dd17896ed91b3dd88300dfb7e6a2fae9a6 not found: ID does not exist"
Oct 30 05:04:39.258085 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:39.258030    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-l8bcz"]
Oct 30 05:04:39.262561 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:39.262533    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-b0fba292-openshift-default-658566db69-l8bcz"]
Oct 30 05:04:39.296242 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:39.296176    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd" podStartSLOduration=1.296158534 podStartE2EDuration="1.296158534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:04:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:04:39.294437466 +0000 UTC m=+1208.518257386" watchObservedRunningTime="2026-10-30 05:04:39.296158534 +0000 UTC m=+1208.519978437"
Oct 30 05:04:39.433727 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:39.433640    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:39.660174 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:39.660147    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd"
Oct 30 05:04:40.157026 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:40.156993    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:40.661383 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:40.661341    5898 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 30 05:04:40.661771 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:40.661413    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd" podUID="736153cf-ff1d-4f3e-b460-a62cd005c988" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 30 05:04:40.907384 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:40.907350    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="860aa2af-bde4-41f8-aa0c-60d3ccaf97a2" path="/var/lib/kubelet/pods/860aa2af-bde4-41f8-aa0c-60d3ccaf97a2/volumes"
Oct 30 05:04:41.660467 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:41.660433    5898 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 30 05:04:41.660641 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:41.660498    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd" podUID="736153cf-ff1d-4f3e-b460-a62cd005c988" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 30 05:04:41.864422 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:41.864390    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:41.957237 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:41.957108    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:42.057422 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:42.057385    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:42.660593 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:42.660554    5898 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 30 05:04:42.660808 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:42.660613    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd" podUID="736153cf-ff1d-4f3e-b460-a62cd005c988" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 30 05:04:43.660187 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:43.660157    5898 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 30 05:04:43.660571 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:43.660217    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd" podUID="736153cf-ff1d-4f3e-b460-a62cd005c988" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 30 05:04:43.760547 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:43.760516    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:44.660648 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:44.660617    5898 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 30 05:04:44.661118 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:44.660669    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd" podUID="736153cf-ff1d-4f3e-b460-a62cd005c988" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 30 05:04:45.660529 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:45.660488    5898 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 30 05:04:45.660831 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:45.660796    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd" podUID="736153cf-ff1d-4f3e-b460-a62cd005c988" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 30 05:04:46.720294 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:46.720222    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:46.824184 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:46.824146    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:46.861200 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:46.861169    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd"
Oct 30 05:04:46.862070 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:46.861762    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd"
Oct 30 05:04:46.862586 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:46.862566    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd"
Oct 30 05:04:47.005995 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:47.001156    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:47.111015 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:47.110980    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:49.264815 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:49.264771    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:50.673457 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:50.673423    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:51.166913 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:51.166879    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:51.601743 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:51.601709    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:51.701833 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:51.701797    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:51.808935 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:51.803560    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:53.336987 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:53.336948    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:53.479943 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:53.479878    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:53.586563 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:53.586524    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:55.584424 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:55.584394    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:56.659273 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:56.659241    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:56.819007 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:04:56.818971    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:04:56.953565 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:56.953485    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:57.724615 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:57.724576    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:57.869403 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:57.869364    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:58.009960 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:58.009854    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:58.815594 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:58.815541    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:58.914811 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:58.914779    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:58.935584 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:58.935554    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e15c99-79bc65cccb-gwzkf"]
Oct 30 05:04:58.936192 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:58.936177    5898 cpu_manager.go:414] "containerMap: removing container" podUID="860aa2af-bde4-41f8-aa0c-60d3ccaf97a2" containerName="istio-proxy"
Oct 30 05:04:58.936239 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:58.936193    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="860aa2af-bde4-41f8-aa0c-60d3ccaf97a2" containerName="istio-proxy"
Oct 30 05:04:58.936302 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:58.936293    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="860aa2af-bde4-41f8-aa0c-60d3ccaf97a2" containerName="istio-proxy"
Oct 30 05:04:58.945071 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:58.945051    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e15c99-79bc65cccb-gwzkf"
Oct 30 05:04:58.957469 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:58.957448    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e15c99-79bc65cccb-gwzkf"]
Oct 30 05:04:58.964180 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:58.964156    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-e15c99\""
Oct 30 05:04:58.964312 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:58.964232    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-e15c99-dockercfg-5ftfk\""
Oct 30 05:04:59.081606 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:59.081567    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:59.116462 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:59.116430    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vws6\" (UniqueName: \"kubernetes.io/projected/ef150e5d-807c-4d9b-9dd0-a3e4953e1e21-kube-api-access-4vws6\") pod \"payload-processing-e2e-shared-b-w3-e15c99-79bc65cccb-gwzkf\" (UID: \"ef150e5d-807c-4d9b-9dd0-a3e4953e1e21\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e15c99-79bc65cccb-gwzkf"
Oct 30 05:04:59.116664 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:59.116648    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef150e5d-807c-4d9b-9dd0-a3e4953e1e21-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-e15c99-79bc65cccb-gwzkf\" (UID: \"ef150e5d-807c-4d9b-9dd0-a3e4953e1e21\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e15c99-79bc65cccb-gwzkf"
Oct 30 05:04:59.225477 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:59.222536    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef150e5d-807c-4d9b-9dd0-a3e4953e1e21-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-e15c99-79bc65cccb-gwzkf\" (UID: \"ef150e5d-807c-4d9b-9dd0-a3e4953e1e21\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e15c99-79bc65cccb-gwzkf"
Oct 30 05:04:59.225477 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:59.222717    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vws6\" (UniqueName: \"kubernetes.io/projected/ef150e5d-807c-4d9b-9dd0-a3e4953e1e21-kube-api-access-4vws6\") pod \"payload-processing-e2e-shared-b-w3-e15c99-79bc65cccb-gwzkf\" (UID: \"ef150e5d-807c-4d9b-9dd0-a3e4953e1e21\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e15c99-79bc65cccb-gwzkf"
Oct 30 05:04:59.225477 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:59.224213    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef150e5d-807c-4d9b-9dd0-a3e4953e1e21-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-e15c99-79bc65cccb-gwzkf\" (UID: \"ef150e5d-807c-4d9b-9dd0-a3e4953e1e21\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e15c99-79bc65cccb-gwzkf"
Oct 30 05:04:59.238626 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:59.238590    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:04:59.277920 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:59.268156    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vws6\" (UniqueName: \"kubernetes.io/projected/ef150e5d-807c-4d9b-9dd0-a3e4953e1e21-kube-api-access-4vws6\") pod \"payload-processing-e2e-shared-b-w3-e15c99-79bc65cccb-gwzkf\" (UID: \"ef150e5d-807c-4d9b-9dd0-a3e4953e1e21\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e15c99-79bc65cccb-gwzkf"
Oct 30 05:04:59.560293 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:59.560216    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e15c99-79bc65cccb-gwzkf"
Oct 30 05:04:59.883863 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:04:59.883831    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e15c99-79bc65cccb-gwzkf"]
Oct 30 05:05:00.231709 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:00.231610    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:00.398776 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:00.398743    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e15c99-79bc65cccb-gwzkf" event={"ID":"ef150e5d-807c-4d9b-9dd0-a3e4953e1e21","Type":"ContainerStarted","Data":"0845d47eb8aafcb7781ef2d5d94330ba9841b4e544f086e0d8a4abd79688283e"}
Oct 30 05:05:00.398982 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:00.398796    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e15c99-79bc65cccb-gwzkf" event={"ID":"ef150e5d-807c-4d9b-9dd0-a3e4953e1e21","Type":"ContainerStarted","Data":"54097021c2cff13df0d974fafadbe836e830774a7ff4148b580e8fe8692ad989"}
Oct 30 05:05:00.398982 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:00.398950    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e15c99-79bc65cccb-gwzkf"
Oct 30 05:05:00.440200 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:00.440137    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e15c99-79bc65cccb-gwzkf" podStartSLOduration=2.440118215 podStartE2EDuration="2.440118215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:04:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:05:00.436704162 +0000 UTC m=+1229.660524084" watchObservedRunningTime="2026-10-30 05:05:00.440118215 +0000 UTC m=+1229.663938120"
Oct 30 05:05:02.087584 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:02.087550    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:03.893776 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:03.893742    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:05.929273 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:05.929241    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:07.719672 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:07.719632    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:09.658247 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:09.658207    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:09.918707 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:09.918617    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:11.026551 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:11.026517    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:11.410444 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:11.410415    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e15c99-79bc65cccb-gwzkf"
Oct 30 05:05:11.600308 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:11.600275    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:12.657701 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:12.657665    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:12.767414 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:12.767373    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:05:12.851498 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:12.843677    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:13.043132 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:13.043044    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c862f32-f4ls6"]
Oct 30 05:05:13.054046 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:13.054013    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c862f32-f4ls6"]
Oct 30 05:05:14.001732 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:14.001697    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:14.890784 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:14.890754    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73011279-ac38-4b5c-a830-6558bb34d850" path="/var/lib/kubelet/pods/73011279-ac38-4b5c-a830-6558bb34d850/volumes"
Oct 30 05:05:15.534104 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:15.534034    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:15.678971 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:15.678932    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:16.264097 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:16.264056    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:18.314308 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:18.314273    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:18.507733 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:18.507698    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:18.702347 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:18.702310    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:18.864967 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:18.863507    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:19.042032 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:19.041922    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:19.268936 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:19.260737    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:19.452587 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:19.452559    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:20.938876 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:20.938827    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:21.397477 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:21.397440    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:22.050933 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:22.050864    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e92d9e6-ctg5b"]
Oct 30 05:05:22.061922 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:22.061870    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e92d9e6-ctg5b"]
Oct 30 05:05:22.893492 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:22.893453    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82219032-2bf7-4bc8-89d3-5063a7ab1f0f" path="/var/lib/kubelet/pods/82219032-2bf7-4bc8-89d3-5063a7ab1f0f/volumes"
Oct 30 05:05:23.134311 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:23.134273    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:23.350805 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:23.350772    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:23.642036 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:23.641888    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:23.866856 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:23.866810    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:25.116999 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:25.116893    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:25.281802 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:25.281768    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-6mdvj"]
Oct 30 05:05:25.286150 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:25.286131    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-6mdvj"
Oct 30 05:05:25.305840 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:25.305818    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-c613f5ea\""
Oct 30 05:05:25.309762 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:25.309737    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:25.357999 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:25.357971    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrzz2\" (UniqueName: \"kubernetes.io/projected/816aafb6-0c1d-419f-9c95-191371977ef4-kube-api-access-hrzz2\") pod \"payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-6mdvj\" (UID: \"816aafb6-0c1d-419f-9c95-191371977ef4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-6mdvj"
Oct 30 05:05:25.358164 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:25.358144    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/816aafb6-0c1d-419f-9c95-191371977ef4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-6mdvj\" (UID: \"816aafb6-0c1d-419f-9c95-191371977ef4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-6mdvj"
Oct 30 05:05:25.444974 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:25.444019    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-6mdvj"]
Oct 30 05:05:25.465925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:25.462475    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/816aafb6-0c1d-419f-9c95-191371977ef4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-6mdvj\" (UID: \"816aafb6-0c1d-419f-9c95-191371977ef4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-6mdvj"
Oct 30 05:05:25.465925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:25.462689    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrzz2\" (UniqueName: \"kubernetes.io/projected/816aafb6-0c1d-419f-9c95-191371977ef4-kube-api-access-hrzz2\") pod \"payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-6mdvj\" (UID: \"816aafb6-0c1d-419f-9c95-191371977ef4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-6mdvj"
Oct 30 05:05:25.485925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:25.467146    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/816aafb6-0c1d-419f-9c95-191371977ef4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-6mdvj\" (UID: \"816aafb6-0c1d-419f-9c95-191371977ef4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-6mdvj"
Oct 30 05:05:25.602926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:25.597339    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrzz2\" (UniqueName: \"kubernetes.io/projected/816aafb6-0c1d-419f-9c95-191371977ef4-kube-api-access-hrzz2\") pod \"payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-6mdvj\" (UID: \"816aafb6-0c1d-419f-9c95-191371977ef4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-6mdvj"
Oct 30 05:05:25.602926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:25.600412    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-6mdvj"
Oct 30 05:05:25.661940 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:25.661888    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:25.679944 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:25.678081    5898 generic.go:356] "Generic (PLEG): container finished" podID="52273f50-a59c-4f49-88cc-1e12c205713a" containerID="9893bf2c15e3500895c24242fd14f0980bc4466805f522c1032b1d9aa6eee5df" exitCode=0
Oct 30 05:05:25.679944 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:25.678387    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk" event={"ID":"52273f50-a59c-4f49-88cc-1e12c205713a","Type":"ContainerDied","Data":"9893bf2c15e3500895c24242fd14f0980bc4466805f522c1032b1d9aa6eee5df"}
Oct 30 05:05:25.679944 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:25.678843    5898 scope.go:122] "RemoveContainer" containerID="9893bf2c15e3500895c24242fd14f0980bc4466805f522c1032b1d9aa6eee5df"
Oct 30 05:05:25.681271 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:25.681110    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:05:25.681271 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:25.681198    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:05:25.681384 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:25.681273    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:05:25.960669 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:25.960641    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-6mdvj"]
Oct 30 05:05:25.961814 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:05:25.961781    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod816aafb6_0c1d_419f_9c95_191371977ef4.slice/crio-b5d6145c642e66b8a7275ee8d5fec726a444699492db898a520da78f751161a1 WatchSource:0}: Error finding container b5d6145c642e66b8a7275ee8d5fec726a444699492db898a520da78f751161a1: Status 404 returned error can't find the container with id b5d6145c642e66b8a7275ee8d5fec726a444699492db898a520da78f751161a1
Oct 30 05:05:26.690674 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:26.690630    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-6mdvj" event={"ID":"816aafb6-0c1d-419f-9c95-191371977ef4","Type":"ContainerStarted","Data":"f99809889cb27825411ba243b1b86eb6caafbb2b34437c8f99b246086bd1a0c9"}
Oct 30 05:05:26.691342 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:26.690702    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-6mdvj" event={"ID":"816aafb6-0c1d-419f-9c95-191371977ef4","Type":"ContainerStarted","Data":"b5d6145c642e66b8a7275ee8d5fec726a444699492db898a520da78f751161a1"}
Oct 30 05:05:26.691342 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:26.690736    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-6mdvj"
Oct 30 05:05:26.693176 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:26.693152    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk" event={"ID":"52273f50-a59c-4f49-88cc-1e12c205713a","Type":"ContainerStarted","Data":"529977aac2d230a0b0aa61eafac3376ee31101765c1d3f68b98144453024d9e7"}
Oct 30 05:05:26.732193 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:26.732148    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-6mdvj" podStartSLOduration=1.732136121 podStartE2EDuration="1.732136121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:05:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:05:26.730441657 +0000 UTC m=+1255.954261558" watchObservedRunningTime="2026-10-30 05:05:26.732136121 +0000 UTC m=+1255.955956021"
Oct 30 05:05:26.819627 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:05:26.819583    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:05:27.318316 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:27.318275    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk"
Oct 30 05:05:27.318523 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:27.318496    5898 patch_prober.go:29] interesting pod/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 30 05:05:27.318575 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:27.318551    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk" podUID="52273f50-a59c-4f49-88cc-1e12c205713a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 30 05:05:27.927121 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:27.927079    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:28.318325 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:28.318236    5898 patch_prober.go:29] interesting pod/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 30 05:05:28.318325 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:28.318292    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk" podUID="52273f50-a59c-4f49-88cc-1e12c205713a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 30 05:05:29.000264 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:29.000232    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:29.129831 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:29.129796    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:29.318419 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:29.318326    5898 patch_prober.go:29] interesting pod/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 30 05:05:29.318419 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:29.318389    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk" podUID="52273f50-a59c-4f49-88cc-1e12c205713a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 30 05:05:29.824564 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:29.824522    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:30.318125 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:30.318036    5898 patch_prober.go:29] interesting pod/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 30 05:05:30.318125 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:30.318110    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk" podUID="52273f50-a59c-4f49-88cc-1e12c205713a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 30 05:05:31.318865 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:31.318831    5898 patch_prober.go:29] interesting pod/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 30 05:05:31.319466 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:31.319433    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk" podUID="52273f50-a59c-4f49-88cc-1e12c205713a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 30 05:05:31.414154 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:31.414111    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:31.462549 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:31.462520    5898 scope.go:122] "RemoveContainer" containerID="c7e908b9b884142eb55a420bd383d652fc30137d2ea9a907bbbcfb43f103ba2e"
Oct 30 05:05:31.481227 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:31.481200    5898 scope.go:122] "RemoveContainer" containerID="4ad7148950bb3ce811c6aee370940875df20b448b3a3064e6e4c1ba75cfa057e"
Oct 30 05:05:31.513672 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:31.513562    5898 scope.go:122] "RemoveContainer" containerID="99c74e91f6bc755caa3754c5fd7dde2a8b4b7319031d1349fcb3c45d2de7c116"
Oct 30 05:05:32.191934 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:32.191306    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:32.319023 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:32.318976    5898 patch_prober.go:29] interesting pod/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 30 05:05:32.319531 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:32.319041    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk" podUID="52273f50-a59c-4f49-88cc-1e12c205713a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 30 05:05:32.490640 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:32.490528    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk"]
Oct 30 05:05:32.491143 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:32.491113    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk" podUID="52273f50-a59c-4f49-88cc-1e12c205713a" containerName="istio-proxy" containerID="cri-o://529977aac2d230a0b0aa61eafac3376ee31101765c1d3f68b98144453024d9e7" gracePeriod=30
Oct 30 05:05:32.499318 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:32.499272    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:32.669004 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:32.668963    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:33.323791 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:33.323761    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:33.873631 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:33.873592    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:34.077926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:34.077552    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:34.936966 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:34.936925    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:35.960184 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:35.960155    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk"
Oct 30 05:05:36.978684 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:36.978650    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:37.090753 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:37.090721    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:37.198292 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:37.198262    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:37.710039 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:37.709988    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:37.713175 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:37.713153    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-6mdvj"
Oct 30 05:05:37.871305 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:37.871251    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:38.056371 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:38.056333    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:38.061174 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:38.061151    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk"
Oct 30 05:05:38.277111 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:38.275399    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52273f50-a59c-4f49-88cc-1e12c205713a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52273f50-a59c-4f49-88cc-1e12c205713a-istio-envoy\") pod \"52273f50-a59c-4f49-88cc-1e12c205713a\" (UID: \"52273f50-a59c-4f49-88cc-1e12c205713a\") "
Oct 30 05:05:38.277111 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:38.275488    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52273f50-a59c-4f49-88cc-1e12c205713a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52273f50-a59c-4f49-88cc-1e12c205713a-istio-data\") pod \"52273f50-a59c-4f49-88cc-1e12c205713a\" (UID: \"52273f50-a59c-4f49-88cc-1e12c205713a\") "
Oct 30 05:05:38.277111 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:38.275599    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52273f50-a59c-4f49-88cc-1e12c205713a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52273f50-a59c-4f49-88cc-1e12c205713a-istiod-ca-cert\") pod \"52273f50-a59c-4f49-88cc-1e12c205713a\" (UID: \"52273f50-a59c-4f49-88cc-1e12c205713a\") "
Oct 30 05:05:38.277111 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:38.275636    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52273f50-a59c-4f49-88cc-1e12c205713a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52273f50-a59c-4f49-88cc-1e12c205713a-workload-socket\") pod \"52273f50-a59c-4f49-88cc-1e12c205713a\" (UID: \"52273f50-a59c-4f49-88cc-1e12c205713a\") "
Oct 30 05:05:38.277111 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:38.275762    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/52273f50-a59c-4f49-88cc-1e12c205713a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52273f50-a59c-4f49-88cc-1e12c205713a-istio-podinfo\") pod \"52273f50-a59c-4f49-88cc-1e12c205713a\" (UID: \"52273f50-a59c-4f49-88cc-1e12c205713a\") "
Oct 30 05:05:38.277111 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:38.275832    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52273f50-a59c-4f49-88cc-1e12c205713a-istio-token\" (UniqueName: \"kubernetes.io/projected/52273f50-a59c-4f49-88cc-1e12c205713a-istio-token\") pod \"52273f50-a59c-4f49-88cc-1e12c205713a\" (UID: \"52273f50-a59c-4f49-88cc-1e12c205713a\") "
Oct 30 05:05:38.277111 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:38.275998    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52273f50-a59c-4f49-88cc-1e12c205713a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52273f50-a59c-4f49-88cc-1e12c205713a-credential-socket\") pod \"52273f50-a59c-4f49-88cc-1e12c205713a\" (UID: \"52273f50-a59c-4f49-88cc-1e12c205713a\") "
Oct 30 05:05:38.277111 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:38.276037    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52273f50-a59c-4f49-88cc-1e12c205713a-kube-api-access-glnsn\" (UniqueName: \"kubernetes.io/projected/52273f50-a59c-4f49-88cc-1e12c205713a-kube-api-access-glnsn\") pod \"52273f50-a59c-4f49-88cc-1e12c205713a\" (UID: \"52273f50-a59c-4f49-88cc-1e12c205713a\") "
Oct 30 05:05:38.277111 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:38.276109    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52273f50-a59c-4f49-88cc-1e12c205713a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52273f50-a59c-4f49-88cc-1e12c205713a-workload-certs\") pod \"52273f50-a59c-4f49-88cc-1e12c205713a\" (UID: \"52273f50-a59c-4f49-88cc-1e12c205713a\") "
Oct 30 05:05:38.278061 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:38.277732    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52273f50-a59c-4f49-88cc-1e12c205713a-workload-certs" pod "52273f50-a59c-4f49-88cc-1e12c205713a" (UID: "52273f50-a59c-4f49-88cc-1e12c205713a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:05:38.279969 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:38.279220    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52273f50-a59c-4f49-88cc-1e12c205713a-istiod-ca-cert" pod "52273f50-a59c-4f49-88cc-1e12c205713a" (UID: "52273f50-a59c-4f49-88cc-1e12c205713a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:05:38.279969 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:38.279446    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52273f50-a59c-4f49-88cc-1e12c205713a-istio-data" pod "52273f50-a59c-4f49-88cc-1e12c205713a" (UID: "52273f50-a59c-4f49-88cc-1e12c205713a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:05:38.280424 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:38.280166    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52273f50-a59c-4f49-88cc-1e12c205713a-credential-socket" pod "52273f50-a59c-4f49-88cc-1e12c205713a" (UID: "52273f50-a59c-4f49-88cc-1e12c205713a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:05:38.281528 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:38.281508    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52273f50-a59c-4f49-88cc-1e12c205713a-workload-socket" pod "52273f50-a59c-4f49-88cc-1e12c205713a" (UID: "52273f50-a59c-4f49-88cc-1e12c205713a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:05:38.314514 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:38.314460    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52273f50-a59c-4f49-88cc-1e12c205713a-istio-podinfo" pod "52273f50-a59c-4f49-88cc-1e12c205713a" (UID: "52273f50-a59c-4f49-88cc-1e12c205713a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:05:38.315711 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:38.315672    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52273f50-a59c-4f49-88cc-1e12c205713a-istio-envoy" pod "52273f50-a59c-4f49-88cc-1e12c205713a" (UID: "52273f50-a59c-4f49-88cc-1e12c205713a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:05:38.315844 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:38.315750    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52273f50-a59c-4f49-88cc-1e12c205713a-kube-api-access-glnsn" pod "52273f50-a59c-4f49-88cc-1e12c205713a" (UID: "52273f50-a59c-4f49-88cc-1e12c205713a"). InnerVolumeSpecName "kube-api-access-glnsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:05:38.317046 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:38.317023    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52273f50-a59c-4f49-88cc-1e12c205713a-istio-token" pod "52273f50-a59c-4f49-88cc-1e12c205713a" (UID: "52273f50-a59c-4f49-88cc-1e12c205713a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:05:38.379704 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:38.379660    5898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52273f50-a59c-4f49-88cc-1e12c205713a-istio-token\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:05:38.379704 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:38.379696    5898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52273f50-a59c-4f49-88cc-1e12c205713a-credential-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:05:38.380012 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:38.379712    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glnsn\" (UniqueName: \"kubernetes.io/projected/52273f50-a59c-4f49-88cc-1e12c205713a-kube-api-access-glnsn\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:05:38.380012 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:38.379727    5898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52273f50-a59c-4f49-88cc-1e12c205713a-workload-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:05:38.380012 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:38.379740    5898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52273f50-a59c-4f49-88cc-1e12c205713a-istio-envoy\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:05:38.380012 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:38.379752    5898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52273f50-a59c-4f49-88cc-1e12c205713a-istio-data\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:05:38.380012 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:38.379766    5898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52273f50-a59c-4f49-88cc-1e12c205713a-istiod-ca-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:05:38.380012 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:38.379778    5898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52273f50-a59c-4f49-88cc-1e12c205713a-workload-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:05:38.380012 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:38.379790    5898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52273f50-a59c-4f49-88cc-1e12c205713a-istio-podinfo\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:05:38.847437 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:38.847402    5898 generic.go:356] "Generic (PLEG): container finished" podID="52273f50-a59c-4f49-88cc-1e12c205713a" containerID="529977aac2d230a0b0aa61eafac3376ee31101765c1d3f68b98144453024d9e7" exitCode=0
Oct 30 05:05:38.847756 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:38.847480    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk" event={"ID":"52273f50-a59c-4f49-88cc-1e12c205713a","Type":"ContainerDied","Data":"529977aac2d230a0b0aa61eafac3376ee31101765c1d3f68b98144453024d9e7"}
Oct 30 05:05:38.847879 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:38.847825    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk" event={"ID":"52273f50-a59c-4f49-88cc-1e12c205713a","Type":"ContainerDied","Data":"f2a8e22f7756a1511d8df0a72700fe6758f058638655d0b0523f08359f6a8f1f"}
Oct 30 05:05:38.847984 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:38.847566    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk"
Oct 30 05:05:38.848045 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:38.847855    5898 scope.go:122] "RemoveContainer" containerID="529977aac2d230a0b0aa61eafac3376ee31101765c1d3f68b98144453024d9e7"
Oct 30 05:05:38.862203 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:38.862179    5898 scope.go:122] "RemoveContainer" containerID="9893bf2c15e3500895c24242fd14f0980bc4466805f522c1032b1d9aa6eee5df"
Oct 30 05:05:38.925921 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:38.925867    5898 scope.go:122] "RemoveContainer" containerID="529977aac2d230a0b0aa61eafac3376ee31101765c1d3f68b98144453024d9e7"
Oct 30 05:05:38.927053 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:05:38.927012    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"529977aac2d230a0b0aa61eafac3376ee31101765c1d3f68b98144453024d9e7\": container with ID starting with 529977aac2d230a0b0aa61eafac3376ee31101765c1d3f68b98144453024d9e7 not found: ID does not exist" containerID="529977aac2d230a0b0aa61eafac3376ee31101765c1d3f68b98144453024d9e7"
Oct 30 05:05:38.927181 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:38.927058    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"529977aac2d230a0b0aa61eafac3376ee31101765c1d3f68b98144453024d9e7"} err="failed to get container status \"529977aac2d230a0b0aa61eafac3376ee31101765c1d3f68b98144453024d9e7\": rpc error: code = NotFound desc = could not find container \"529977aac2d230a0b0aa61eafac3376ee31101765c1d3f68b98144453024d9e7\": container with ID starting with 529977aac2d230a0b0aa61eafac3376ee31101765c1d3f68b98144453024d9e7 not found: ID does not exist"
Oct 30 05:05:38.927181 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:38.927083    5898 scope.go:122] "RemoveContainer" containerID="9893bf2c15e3500895c24242fd14f0980bc4466805f522c1032b1d9aa6eee5df"
Oct 30 05:05:38.927407 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:05:38.927374    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9893bf2c15e3500895c24242fd14f0980bc4466805f522c1032b1d9aa6eee5df\": container with ID starting with 9893bf2c15e3500895c24242fd14f0980bc4466805f522c1032b1d9aa6eee5df not found: ID does not exist" containerID="9893bf2c15e3500895c24242fd14f0980bc4466805f522c1032b1d9aa6eee5df"
Oct 30 05:05:38.927491 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:38.927411    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9893bf2c15e3500895c24242fd14f0980bc4466805f522c1032b1d9aa6eee5df"} err="failed to get container status \"9893bf2c15e3500895c24242fd14f0980bc4466805f522c1032b1d9aa6eee5df\": rpc error: code = NotFound desc = could not find container \"9893bf2c15e3500895c24242fd14f0980bc4466805f522c1032b1d9aa6eee5df\": container with ID starting with 9893bf2c15e3500895c24242fd14f0980bc4466805f522c1032b1d9aa6eee5df not found: ID does not exist"
Oct 30 05:05:38.950834 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:38.950808    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk"]
Oct 30 05:05:38.954960 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:38.954934    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-5a926a9d-gw-openshift-default-65b77bf777-b7gbk"]
Oct 30 05:05:39.606925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:39.606327    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:39.823432 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:39.823395    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:39.939847 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:39.939556    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:40.042543 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:40.042510    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:40.155275 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:40.155236    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:40.333135 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:40.333093    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:40.582706 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:40.582665    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-44bc69-59d6bdddbc-c58st"]
Oct 30 05:05:40.583700 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:40.583623    5898 cpu_manager.go:414] "containerMap: removing container" podUID="52273f50-a59c-4f49-88cc-1e12c205713a" containerName="istio-proxy"
Oct 30 05:05:40.583700 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:40.583659    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52273f50-a59c-4f49-88cc-1e12c205713a" containerName="istio-proxy"
Oct 30 05:05:40.583892 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:40.583867    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="52273f50-a59c-4f49-88cc-1e12c205713a" containerName="istio-proxy"
Oct 30 05:05:40.583990 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:40.583891    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="52273f50-a59c-4f49-88cc-1e12c205713a" containerName="istio-proxy"
Oct 30 05:05:40.627104 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:40.627071    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-44bc69-59d6bdddbc-c58st"]
Oct 30 05:05:40.627774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:40.627750    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-44bc69-59d6bdddbc-c58st"
Oct 30 05:05:40.678924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:40.659706    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfp9t\" (UniqueName: \"kubernetes.io/projected/fc35e0f4-fbc7-43e8-bf45-10d4d3ff19fe-kube-api-access-sfp9t\") pod \"payload-pre-processing-e2e-shared-a-w2-44bc69-59d6bdddbc-c58st\" (UID: \"fc35e0f4-fbc7-43e8-bf45-10d4d3ff19fe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-44bc69-59d6bdddbc-c58st"
Oct 30 05:05:40.678924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:40.659996    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc35e0f4-fbc7-43e8-bf45-10d4d3ff19fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-44bc69-59d6bdddbc-c58st\" (UID: \"fc35e0f4-fbc7-43e8-bf45-10d4d3ff19fe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-44bc69-59d6bdddbc-c58st"
Oct 30 05:05:40.824118 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:40.824084    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-44bc69-5784cbb69b-68q6q"]
Oct 30 05:05:40.828887 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:40.828859    5898 cpu_manager.go:414] "containerMap: removing container" podUID="52273f50-a59c-4f49-88cc-1e12c205713a" containerName="istio-proxy"
Oct 30 05:05:40.846057 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:40.843998    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52273f50-a59c-4f49-88cc-1e12c205713a" containerName="istio-proxy"
Oct 30 05:05:40.849047 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:40.829317    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfp9t\" (UniqueName: \"kubernetes.io/projected/fc35e0f4-fbc7-43e8-bf45-10d4d3ff19fe-kube-api-access-sfp9t\") pod \"payload-pre-processing-e2e-shared-a-w2-44bc69-59d6bdddbc-c58st\" (UID: \"fc35e0f4-fbc7-43e8-bf45-10d4d3ff19fe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-44bc69-59d6bdddbc-c58st"
Oct 30 05:05:40.849047 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:40.846442    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-44bc69\""
Oct 30 05:05:40.849047 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:40.846789    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc35e0f4-fbc7-43e8-bf45-10d4d3ff19fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-44bc69-59d6bdddbc-c58st\" (UID: \"fc35e0f4-fbc7-43e8-bf45-10d4d3ff19fe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-44bc69-59d6bdddbc-c58st"
Oct 30 05:05:40.849047 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:40.847939    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc35e0f4-fbc7-43e8-bf45-10d4d3ff19fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-44bc69-59d6bdddbc-c58st\" (UID: \"fc35e0f4-fbc7-43e8-bf45-10d4d3ff19fe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-44bc69-59d6bdddbc-c58st"
Oct 30 05:05:40.849047 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:40.848679    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-44bc69-5784cbb69b-68q6q"
Oct 30 05:05:40.849047 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:40.824484    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-44bc69-dockercfg-db8rj\""
Oct 30 05:05:40.861669 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:40.861648    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-44bc69-5784cbb69b-68q6q"]
Oct 30 05:05:40.878997 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:40.873947    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfp9t\" (UniqueName: \"kubernetes.io/projected/fc35e0f4-fbc7-43e8-bf45-10d4d3ff19fe-kube-api-access-sfp9t\") pod \"payload-pre-processing-e2e-shared-a-w2-44bc69-59d6bdddbc-c58st\" (UID: \"fc35e0f4-fbc7-43e8-bf45-10d4d3ff19fe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-44bc69-59d6bdddbc-c58st"
Oct 30 05:05:40.880422 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:40.880384    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-44bc69-59d6bdddbc-c58st"
Oct 30 05:05:40.902693 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:40.902661    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52273f50-a59c-4f49-88cc-1e12c205713a" path="/var/lib/kubelet/pods/52273f50-a59c-4f49-88cc-1e12c205713a/volumes"
Oct 30 05:05:40.948212 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:40.948024    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq2gf\" (UniqueName: \"kubernetes.io/projected/3270ef0e-4775-48c6-8724-baff8e3fac06-kube-api-access-fq2gf\") pod \"payload-processing-e2e-shared-a-w2-44bc69-5784cbb69b-68q6q\" (UID: \"3270ef0e-4775-48c6-8724-baff8e3fac06\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-44bc69-5784cbb69b-68q6q"
Oct 30 05:05:40.948647 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:40.948589    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3270ef0e-4775-48c6-8724-baff8e3fac06-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-44bc69-5784cbb69b-68q6q\" (UID: \"3270ef0e-4775-48c6-8724-baff8e3fac06\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-44bc69-5784cbb69b-68q6q"
Oct 30 05:05:41.053933 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:41.051412    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq2gf\" (UniqueName: \"kubernetes.io/projected/3270ef0e-4775-48c6-8724-baff8e3fac06-kube-api-access-fq2gf\") pod \"payload-processing-e2e-shared-a-w2-44bc69-5784cbb69b-68q6q\" (UID: \"3270ef0e-4775-48c6-8724-baff8e3fac06\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-44bc69-5784cbb69b-68q6q"
Oct 30 05:05:41.053933 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:41.051500    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3270ef0e-4775-48c6-8724-baff8e3fac06-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-44bc69-5784cbb69b-68q6q\" (UID: \"3270ef0e-4775-48c6-8724-baff8e3fac06\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-44bc69-5784cbb69b-68q6q"
Oct 30 05:05:41.053933 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:41.052881    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3270ef0e-4775-48c6-8724-baff8e3fac06-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-44bc69-5784cbb69b-68q6q\" (UID: \"3270ef0e-4775-48c6-8724-baff8e3fac06\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-44bc69-5784cbb69b-68q6q"
Oct 30 05:05:41.096688 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:41.096429    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq2gf\" (UniqueName: \"kubernetes.io/projected/3270ef0e-4775-48c6-8724-baff8e3fac06-kube-api-access-fq2gf\") pod \"payload-processing-e2e-shared-a-w2-44bc69-5784cbb69b-68q6q\" (UID: \"3270ef0e-4775-48c6-8724-baff8e3fac06\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-44bc69-5784cbb69b-68q6q"
Oct 30 05:05:41.245611 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:41.244577    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-44bc69-5784cbb69b-68q6q"
Oct 30 05:05:41.312198 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:05:41.311262    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc35e0f4_fbc7_43e8_bf45_10d4d3ff19fe.slice/crio-260e3379a1a3147210e722fde71dcdda33828a3651a56531c53c0a6e92d77b43 WatchSource:0}: Error finding container 260e3379a1a3147210e722fde71dcdda33828a3651a56531c53c0a6e92d77b43: Status 404 returned error can't find the container with id 260e3379a1a3147210e722fde71dcdda33828a3651a56531c53c0a6e92d77b43
Oct 30 05:05:41.318692 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:41.318607    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-44bc69-59d6bdddbc-c58st"]
Oct 30 05:05:41.570289 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:05:41.570257    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3270ef0e_4775_48c6_8724_baff8e3fac06.slice/crio-1ed60c8b57a93ac85f7d85d5a074ae4b9e55bf41fe53514ceb8275d23a06c1cc WatchSource:0}: Error finding container 1ed60c8b57a93ac85f7d85d5a074ae4b9e55bf41fe53514ceb8275d23a06c1cc: Status 404 returned error can't find the container with id 1ed60c8b57a93ac85f7d85d5a074ae4b9e55bf41fe53514ceb8275d23a06c1cc
Oct 30 05:05:41.570678 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:41.570658    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-44bc69-5784cbb69b-68q6q"]
Oct 30 05:05:41.880451 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:41.880350    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-44bc69-5784cbb69b-68q6q" event={"ID":"3270ef0e-4775-48c6-8724-baff8e3fac06","Type":"ContainerStarted","Data":"c84467b0e29b66363e3614465fd70c9845a14d26565a777c18932e81d947f6a6"}
Oct 30 05:05:41.880887 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:41.880453    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-44bc69-5784cbb69b-68q6q" event={"ID":"3270ef0e-4775-48c6-8724-baff8e3fac06","Type":"ContainerStarted","Data":"1ed60c8b57a93ac85f7d85d5a074ae4b9e55bf41fe53514ceb8275d23a06c1cc"}
Oct 30 05:05:41.880887 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:41.880484    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-44bc69-5784cbb69b-68q6q"
Oct 30 05:05:41.882389 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:41.882360    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-44bc69-59d6bdddbc-c58st" event={"ID":"fc35e0f4-fbc7-43e8-bf45-10d4d3ff19fe","Type":"ContainerStarted","Data":"c3469c41e58626e4c573f5c2be26f18bcb97261e12f136cb68c7002bc6f24893"}
Oct 30 05:05:41.882517 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:41.882409    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-44bc69-59d6bdddbc-c58st" event={"ID":"fc35e0f4-fbc7-43e8-bf45-10d4d3ff19fe","Type":"ContainerStarted","Data":"260e3379a1a3147210e722fde71dcdda33828a3651a56531c53c0a6e92d77b43"}
Oct 30 05:05:41.882517 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:41.882483    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-44bc69-59d6bdddbc-c58st"
Oct 30 05:05:41.968831 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:41.968774    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-44bc69-5784cbb69b-68q6q" podStartSLOduration=1.968759878 podStartE2EDuration="1.968759878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:05:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:05:41.965256077 +0000 UTC m=+1271.189075996" watchObservedRunningTime="2026-10-30 05:05:41.968759878 +0000 UTC m=+1271.192579778"
Oct 30 05:05:42.030404 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:42.030319    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-44bc69-59d6bdddbc-c58st" podStartSLOduration=2.030303633 podStartE2EDuration="2.030303633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:05:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:05:42.029055392 +0000 UTC m=+1271.252875324" watchObservedRunningTime="2026-10-30 05:05:42.030303633 +0000 UTC m=+1271.254123536"
Oct 30 05:05:42.283972 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:42.280716    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:44.239098 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:44.239062    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:46.330132 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:46.330095    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:47.769098 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:47.769059    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:48.200830 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:48.200797    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:48.376438 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:48.376405    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:48.819718 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:48.819685    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:48.990634 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:48.990596    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:49.263126 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:49.263097    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:49.442990 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:49.442960    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:51.173692 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:51.173650    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:52.110407 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:52.110370    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:52.390972 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:52.390841    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:52.615986 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:52.615950    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:52.828715 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:52.828667    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:52.897166 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:52.897135    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-44bc69-59d6bdddbc-c58st"
Oct 30 05:05:52.897374 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:52.897208    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-44bc69-5784cbb69b-68q6q"
Oct 30 05:05:53.588237 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:53.588191    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:54.636575 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:54.636539    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:55.065215 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:55.065128    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:55.217141 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:55.217105    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:56.820090 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:05:56.820049    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:05:57.270134 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:57.270100    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:58.479507 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:58.479469    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:58.632875 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:58.632827    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:05:59.729795 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:05:59.729754    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:00.144544 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:00.144515    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:00.459622 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:00.459506    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:00.590139 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:00.590105    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ff4760-78747f9bc9-hwqpl"]
Oct 30 05:06:00.594535 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:00.594514    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ff4760-78747f9bc9-hwqpl"
Oct 30 05:06:00.620657 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:00.620635    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-ff4760\""
Oct 30 05:06:00.634565 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:00.634537    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ff4760-78747f9bc9-hwqpl"]
Oct 30 05:06:00.694418 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:00.694380    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkfq2\" (UniqueName: \"kubernetes.io/projected/7063ee78-5b1b-4f4a-b1f0-7898af800db1-kube-api-access-kkfq2\") pod \"payload-pre-processing-e2e-shared-a-w0-ff4760-78747f9bc9-hwqpl\" (UID: \"7063ee78-5b1b-4f4a-b1f0-7898af800db1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ff4760-78747f9bc9-hwqpl"
Oct 30 05:06:00.694607 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:00.694591    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7063ee78-5b1b-4f4a-b1f0-7898af800db1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-ff4760-78747f9bc9-hwqpl\" (UID: \"7063ee78-5b1b-4f4a-b1f0-7898af800db1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ff4760-78747f9bc9-hwqpl"
Oct 30 05:06:00.731607 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:00.731499    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:00.797934 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:00.797875    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7063ee78-5b1b-4f4a-b1f0-7898af800db1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-ff4760-78747f9bc9-hwqpl\" (UID: \"7063ee78-5b1b-4f4a-b1f0-7898af800db1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ff4760-78747f9bc9-hwqpl"
Oct 30 05:06:00.809944 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:00.809893    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkfq2\" (UniqueName: \"kubernetes.io/projected/7063ee78-5b1b-4f4a-b1f0-7898af800db1-kube-api-access-kkfq2\") pod \"payload-pre-processing-e2e-shared-a-w0-ff4760-78747f9bc9-hwqpl\" (UID: \"7063ee78-5b1b-4f4a-b1f0-7898af800db1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ff4760-78747f9bc9-hwqpl"
Oct 30 05:06:00.810890 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:00.799583    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7063ee78-5b1b-4f4a-b1f0-7898af800db1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-ff4760-78747f9bc9-hwqpl\" (UID: \"7063ee78-5b1b-4f4a-b1f0-7898af800db1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ff4760-78747f9bc9-hwqpl"
Oct 30 05:06:00.850373 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:00.850302    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkfq2\" (UniqueName: \"kubernetes.io/projected/7063ee78-5b1b-4f4a-b1f0-7898af800db1-kube-api-access-kkfq2\") pod \"payload-pre-processing-e2e-shared-a-w0-ff4760-78747f9bc9-hwqpl\" (UID: \"7063ee78-5b1b-4f4a-b1f0-7898af800db1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ff4760-78747f9bc9-hwqpl"
Oct 30 05:06:00.917941 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:00.917892    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ff4760-78747f9bc9-hwqpl"
Oct 30 05:06:01.269966 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:01.269937    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ff4760-78747f9bc9-hwqpl"]
Oct 30 05:06:01.271845 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:06:01.271820    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7063ee78_5b1b_4f4a_b1f0_7898af800db1.slice/crio-23ecb3dc2afb2a47c0b4f1504fae3f5628c2179aa77c843c5138a388f15ce34c WatchSource:0}: Error finding container 23ecb3dc2afb2a47c0b4f1504fae3f5628c2179aa77c843c5138a388f15ce34c: Status 404 returned error can't find the container with id 23ecb3dc2afb2a47c0b4f1504fae3f5628c2179aa77c843c5138a388f15ce34c
Oct 30 05:06:02.038576 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:02.038534    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ff4760-78747f9bc9-hwqpl" event={"ID":"7063ee78-5b1b-4f4a-b1f0-7898af800db1","Type":"ContainerStarted","Data":"e9868171209a0cafc17670b186989bc32023e1ea90a6b4c06f133f3fa9960bfa"}
Oct 30 05:06:02.039007 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:02.038602    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ff4760-78747f9bc9-hwqpl" event={"ID":"7063ee78-5b1b-4f4a-b1f0-7898af800db1","Type":"ContainerStarted","Data":"23ecb3dc2afb2a47c0b4f1504fae3f5628c2179aa77c843c5138a388f15ce34c"}
Oct 30 05:06:02.039007 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:02.038631    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ff4760-78747f9bc9-hwqpl"
Oct 30 05:06:02.095552 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:02.095502    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ff4760-78747f9bc9-hwqpl" podStartSLOduration=2.095487887 podStartE2EDuration="2.095487887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:06:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:06:02.092723915 +0000 UTC m=+1291.316543816" watchObservedRunningTime="2026-10-30 05:06:02.095487887 +0000 UTC m=+1291.319307797"
Oct 30 05:06:02.207178 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:02.207149    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:04.353132 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:04.353094    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:04.508731 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:04.508696    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:06.399677 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:06.399632    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:08.193584 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:08.193544    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:10.157212 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:10.157177    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:11.476293 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:11.474110    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:06:12.091791 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:12.091752    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:13.049570 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:13.049540    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ff4760-78747f9bc9-hwqpl"
Oct 30 05:06:14.229419 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:14.229388    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:16.184830 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:16.184800    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:18.023041 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:18.023001    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:19.360503 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:19.360472    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:19.971060 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:19.971023    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:21.755160 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:21.755098    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:21.804149 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:21.804117    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ff4760-7d4b656bc8-55s9p"]
Oct 30 05:06:21.808413 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:21.808395    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ff4760-7d4b656bc8-55s9p"
Oct 30 05:06:21.847725 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:21.847170    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-ff4760-serving-cert\""
Oct 30 05:06:21.854667 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:21.854639    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ff4760-7d4b656bc8-55s9p"]
Oct 30 05:06:21.942830 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:21.942796    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfd4e937-c236-4f33-8477-0c55aa536d30-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-ff4760-7d4b656bc8-55s9p\" (UID: \"dfd4e937-c236-4f33-8477-0c55aa536d30\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ff4760-7d4b656bc8-55s9p"
Oct 30 05:06:21.943007 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:21.942976    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfd4e937-c236-4f33-8477-0c55aa536d30-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-ff4760-7d4b656bc8-55s9p\" (UID: \"dfd4e937-c236-4f33-8477-0c55aa536d30\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ff4760-7d4b656bc8-55s9p"
Oct 30 05:06:21.943112 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:21.943093    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmj99\" (UniqueName: \"kubernetes.io/projected/dfd4e937-c236-4f33-8477-0c55aa536d30-kube-api-access-vmj99\") pod \"maas-api-e2e-shared-a-w0-ff4760-7d4b656bc8-55s9p\" (UID: \"dfd4e937-c236-4f33-8477-0c55aa536d30\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ff4760-7d4b656bc8-55s9p"
Oct 30 05:06:21.943180 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:21.943166    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfd4e937-c236-4f33-8477-0c55aa536d30-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-ff4760-7d4b656bc8-55s9p\" (UID: \"dfd4e937-c236-4f33-8477-0c55aa536d30\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ff4760-7d4b656bc8-55s9p"
Oct 30 05:06:21.987108 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:21.987040    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:22.044426 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:22.044350    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmj99\" (UniqueName: \"kubernetes.io/projected/dfd4e937-c236-4f33-8477-0c55aa536d30-kube-api-access-vmj99\") pod \"maas-api-e2e-shared-a-w0-ff4760-7d4b656bc8-55s9p\" (UID: \"dfd4e937-c236-4f33-8477-0c55aa536d30\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ff4760-7d4b656bc8-55s9p"
Oct 30 05:06:22.044426 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:22.044404    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfd4e937-c236-4f33-8477-0c55aa536d30-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-ff4760-7d4b656bc8-55s9p\" (UID: \"dfd4e937-c236-4f33-8477-0c55aa536d30\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ff4760-7d4b656bc8-55s9p"
Oct 30 05:06:22.044597 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:22.044572    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfd4e937-c236-4f33-8477-0c55aa536d30-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-ff4760-7d4b656bc8-55s9p\" (UID: \"dfd4e937-c236-4f33-8477-0c55aa536d30\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ff4760-7d4b656bc8-55s9p"
Oct 30 05:06:22.044728 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:22.044699    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfd4e937-c236-4f33-8477-0c55aa536d30-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-ff4760-7d4b656bc8-55s9p\" (UID: \"dfd4e937-c236-4f33-8477-0c55aa536d30\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ff4760-7d4b656bc8-55s9p"
Oct 30 05:06:22.045346 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:22.045322    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfd4e937-c236-4f33-8477-0c55aa536d30-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-ff4760-7d4b656bc8-55s9p\" (UID: \"dfd4e937-c236-4f33-8477-0c55aa536d30\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ff4760-7d4b656bc8-55s9p"
Oct 30 05:06:22.047170 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:22.047146    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfd4e937-c236-4f33-8477-0c55aa536d30-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-ff4760-7d4b656bc8-55s9p\" (UID: \"dfd4e937-c236-4f33-8477-0c55aa536d30\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ff4760-7d4b656bc8-55s9p"
Oct 30 05:06:22.047365 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:22.047345    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfd4e937-c236-4f33-8477-0c55aa536d30-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-ff4760-7d4b656bc8-55s9p\" (UID: \"dfd4e937-c236-4f33-8477-0c55aa536d30\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ff4760-7d4b656bc8-55s9p"
Oct 30 05:06:22.065896 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:22.065870    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmj99\" (UniqueName: \"kubernetes.io/projected/dfd4e937-c236-4f33-8477-0c55aa536d30-kube-api-access-vmj99\") pod \"maas-api-e2e-shared-a-w0-ff4760-7d4b656bc8-55s9p\" (UID: \"dfd4e937-c236-4f33-8477-0c55aa536d30\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ff4760-7d4b656bc8-55s9p"
Oct 30 05:06:22.132929 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:22.129187    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ff4760-7d4b656bc8-55s9p"
Oct 30 05:06:22.143764 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:22.143727    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:22.481748 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:22.481719    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ff4760-7d4b656bc8-55s9p"]
Oct 30 05:06:22.483047 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:06:22.483007    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfd4e937_c236_4f33_8477_0c55aa536d30.slice/crio-39f20a20e2c739b5fa668f1625007c73bb659ed87e21d789bb34ea864f5a8183 WatchSource:0}: Error finding container 39f20a20e2c739b5fa668f1625007c73bb659ed87e21d789bb34ea864f5a8183: Status 404 returned error can't find the container with id 39f20a20e2c739b5fa668f1625007c73bb659ed87e21d789bb34ea864f5a8183
Oct 30 05:06:23.219334 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:23.219295    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ff4760-7d4b656bc8-55s9p" event={"ID":"dfd4e937-c236-4f33-8477-0c55aa536d30","Type":"ContainerStarted","Data":"ad814148294ed35b05da2c30180d82dc46f2e3f477f1c83eb1b9a731fb40f256"}
Oct 30 05:06:23.219827 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:23.219357    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ff4760-7d4b656bc8-55s9p" event={"ID":"dfd4e937-c236-4f33-8477-0c55aa536d30","Type":"ContainerStarted","Data":"39f20a20e2c739b5fa668f1625007c73bb659ed87e21d789bb34ea864f5a8183"}
Oct 30 05:06:23.219827 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:23.219384    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ff4760-7d4b656bc8-55s9p"
Oct 30 05:06:23.264520 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:23.264457    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ff4760-7d4b656bc8-55s9p" podStartSLOduration=1.969433542 podStartE2EDuration="2.264439241s" totalImagesPullingTime="295.005699ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:06:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:06:23.258965691 +0000 UTC m=+1312.482785592" watchObservedRunningTime="2026-10-30 05:06:23.264439241 +0000 UTC m=+1312.488259143"
Oct 30 05:06:23.772716 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:23.772678    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-c613f5ea/e2e-lifecycle-model-c613f5ea-kserve-5bf54d4c45-n5286"]
Oct 30 05:06:23.778494 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:23.778466    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c613f5ea/e2e-lifecycle-model-c613f5ea-kserve-5bf54d4c45-n5286"
Oct 30 05:06:23.796973 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:23.796946    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-c613f5ea/e2e-lifecycle-model-c613f5ea-kserve-5bf54d4c45-n5286"]
Oct 30 05:06:23.804851 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:23.804823    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-c613f5ea\"/\"e2e-lifecycle-model-c613f5ea-kserve-self-signed-certs\""
Oct 30 05:06:23.805049 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:23.804865    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-c613f5ea\"/\"openshift-service-ca.crt\""
Oct 30 05:06:23.805049 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:23.804890    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-c613f5ea\"/\"default-dockercfg-gwzp9\""
Oct 30 05:06:23.805343 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:23.805320    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-c613f5ea\"/\"kube-root-ca.crt\""
Oct 30 05:06:23.811678 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:23.811639    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:23.888769 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:23.888730    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f-tls-certs\") pod \"e2e-lifecycle-model-c613f5ea-kserve-5bf54d4c45-n5286\" (UID: \"ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f\") " pod="ai-tenant-e2e-mt-c613f5ea/e2e-lifecycle-model-c613f5ea-kserve-5bf54d4c45-n5286"
Oct 30 05:06:23.888952 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:23.888873    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f-tmp-dir\") pod \"e2e-lifecycle-model-c613f5ea-kserve-5bf54d4c45-n5286\" (UID: \"ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f\") " pod="ai-tenant-e2e-mt-c613f5ea/e2e-lifecycle-model-c613f5ea-kserve-5bf54d4c45-n5286"
Oct 30 05:06:23.888952 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:23.888933    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f-dshm\") pod \"e2e-lifecycle-model-c613f5ea-kserve-5bf54d4c45-n5286\" (UID: \"ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f\") " pod="ai-tenant-e2e-mt-c613f5ea/e2e-lifecycle-model-c613f5ea-kserve-5bf54d4c45-n5286"
Oct 30 05:06:23.889040 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:23.889015    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f-home\") pod \"e2e-lifecycle-model-c613f5ea-kserve-5bf54d4c45-n5286\" (UID: \"ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f\") " pod="ai-tenant-e2e-mt-c613f5ea/e2e-lifecycle-model-c613f5ea-kserve-5bf54d4c45-n5286"
Oct 30 05:06:23.889073 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:23.889060    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vgft\" (UniqueName: \"kubernetes.io/projected/ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f-kube-api-access-9vgft\") pod \"e2e-lifecycle-model-c613f5ea-kserve-5bf54d4c45-n5286\" (UID: \"ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f\") " pod="ai-tenant-e2e-mt-c613f5ea/e2e-lifecycle-model-c613f5ea-kserve-5bf54d4c45-n5286"
Oct 30 05:06:23.889215 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:23.889193    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f-model-cache\") pod \"e2e-lifecycle-model-c613f5ea-kserve-5bf54d4c45-n5286\" (UID: \"ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f\") " pod="ai-tenant-e2e-mt-c613f5ea/e2e-lifecycle-model-c613f5ea-kserve-5bf54d4c45-n5286"
Oct 30 05:06:23.990957 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:23.990891    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f-model-cache\") pod \"e2e-lifecycle-model-c613f5ea-kserve-5bf54d4c45-n5286\" (UID: \"ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f\") " pod="ai-tenant-e2e-mt-c613f5ea/e2e-lifecycle-model-c613f5ea-kserve-5bf54d4c45-n5286"
Oct 30 05:06:23.991145 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:23.991095    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f-tls-certs\") pod \"e2e-lifecycle-model-c613f5ea-kserve-5bf54d4c45-n5286\" (UID: \"ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f\") " pod="ai-tenant-e2e-mt-c613f5ea/e2e-lifecycle-model-c613f5ea-kserve-5bf54d4c45-n5286"
Oct 30 05:06:23.991243 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:23.991223    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f-tmp-dir\") pod \"e2e-lifecycle-model-c613f5ea-kserve-5bf54d4c45-n5286\" (UID: \"ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f\") " pod="ai-tenant-e2e-mt-c613f5ea/e2e-lifecycle-model-c613f5ea-kserve-5bf54d4c45-n5286"
Oct 30 05:06:23.991394 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:23.991270    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f-dshm\") pod \"e2e-lifecycle-model-c613f5ea-kserve-5bf54d4c45-n5286\" (UID: \"ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f\") " pod="ai-tenant-e2e-mt-c613f5ea/e2e-lifecycle-model-c613f5ea-kserve-5bf54d4c45-n5286"
Oct 30 05:06:23.991394 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:23.991307    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f-model-cache\") pod \"e2e-lifecycle-model-c613f5ea-kserve-5bf54d4c45-n5286\" (UID: \"ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f\") " pod="ai-tenant-e2e-mt-c613f5ea/e2e-lifecycle-model-c613f5ea-kserve-5bf54d4c45-n5286"
Oct 30 05:06:23.991557 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:23.991411    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f-home\") pod \"e2e-lifecycle-model-c613f5ea-kserve-5bf54d4c45-n5286\" (UID: \"ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f\") " pod="ai-tenant-e2e-mt-c613f5ea/e2e-lifecycle-model-c613f5ea-kserve-5bf54d4c45-n5286"
Oct 30 05:06:23.991557 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:23.991489    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vgft\" (UniqueName: \"kubernetes.io/projected/ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f-kube-api-access-9vgft\") pod \"e2e-lifecycle-model-c613f5ea-kserve-5bf54d4c45-n5286\" (UID: \"ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f\") " pod="ai-tenant-e2e-mt-c613f5ea/e2e-lifecycle-model-c613f5ea-kserve-5bf54d4c45-n5286"
Oct 30 05:06:23.991920 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:23.991874    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f-home\") pod \"e2e-lifecycle-model-c613f5ea-kserve-5bf54d4c45-n5286\" (UID: \"ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f\") " pod="ai-tenant-e2e-mt-c613f5ea/e2e-lifecycle-model-c613f5ea-kserve-5bf54d4c45-n5286"
Oct 30 05:06:23.992036 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:23.991917    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f-tmp-dir\") pod \"e2e-lifecycle-model-c613f5ea-kserve-5bf54d4c45-n5286\" (UID: \"ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f\") " pod="ai-tenant-e2e-mt-c613f5ea/e2e-lifecycle-model-c613f5ea-kserve-5bf54d4c45-n5286"
Oct 30 05:06:23.993779 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:23.993756    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f-dshm\") pod \"e2e-lifecycle-model-c613f5ea-kserve-5bf54d4c45-n5286\" (UID: \"ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f\") " pod="ai-tenant-e2e-mt-c613f5ea/e2e-lifecycle-model-c613f5ea-kserve-5bf54d4c45-n5286"
Oct 30 05:06:23.994037 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:23.994015    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f-tls-certs\") pod \"e2e-lifecycle-model-c613f5ea-kserve-5bf54d4c45-n5286\" (UID: \"ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f\") " pod="ai-tenant-e2e-mt-c613f5ea/e2e-lifecycle-model-c613f5ea-kserve-5bf54d4c45-n5286"
Oct 30 05:06:24.055968 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:24.055207    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vgft\" (UniqueName: \"kubernetes.io/projected/ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f-kube-api-access-9vgft\") pod \"e2e-lifecycle-model-c613f5ea-kserve-5bf54d4c45-n5286\" (UID: \"ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f\") " pod="ai-tenant-e2e-mt-c613f5ea/e2e-lifecycle-model-c613f5ea-kserve-5bf54d4c45-n5286"
Oct 30 05:06:24.065741 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:24.065300    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:24.104923 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:24.098098    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c613f5ea/e2e-lifecycle-model-c613f5ea-kserve-5bf54d4c45-n5286"
Oct 30 05:06:24.509260 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:24.507971    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-c613f5ea/e2e-lifecycle-model-c613f5ea-kserve-5bf54d4c45-n5286"]
Oct 30 05:06:24.509260 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:06:24.508061    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec2a0e9c_64ab_4b68_8caf_20c9f8f9126f.slice/crio-a86105a94a7f00fa768021c694fc72f83443982bebad767330aca895dc56087e WatchSource:0}: Error finding container a86105a94a7f00fa768021c694fc72f83443982bebad767330aca895dc56087e: Status 404 returned error can't find the container with id a86105a94a7f00fa768021c694fc72f83443982bebad767330aca895dc56087e
Oct 30 05:06:25.243966 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:25.243932    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c613f5ea/e2e-lifecycle-model-c613f5ea-kserve-5bf54d4c45-n5286" event={"ID":"ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f","Type":"ContainerStarted","Data":"163ac35174564bc849bd44b7c142c921a2c8196ea0ba425090f7d507507e9057"}
Oct 30 05:06:25.244174 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:25.243981    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c613f5ea/e2e-lifecycle-model-c613f5ea-kserve-5bf54d4c45-n5286" event={"ID":"ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f","Type":"ContainerStarted","Data":"a86105a94a7f00fa768021c694fc72f83443982bebad767330aca895dc56087e"}
Oct 30 05:06:25.284703 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:25.284647    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-c613f5ea/e2e-lifecycle-model-c613f5ea-kserve-5bf54d4c45-n5286" podStartSLOduration=2.28463012 podStartE2EDuration="2.28463012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:06:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:06:25.283308311 +0000 UTC m=+1314.507128212" watchObservedRunningTime="2026-10-30 05:06:25.28463012 +0000 UTC m=+1314.508450030"
Oct 30 05:06:25.878175 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:25.878112    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:26.099380 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:26.099346    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-c613f5ea/e2e-lifecycle-model-c613f5ea-kserve-5bf54d4c45-n5286"
Oct 30 05:06:26.821230 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:06:26.821191    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:06:27.833240 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:27.833199    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:29.163273 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:29.163242    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:29.242572 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:29.242546    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ff4760-7d4b656bc8-55s9p"
Oct 30 05:06:29.719163 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:29.719119    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:31.677652 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:31.677621    5898 scope.go:122] "RemoveContainer" containerID="1b7e6b22c423f1001c5dde6b568377210f1501c670537b2484c29dc90102a0b0"
Oct 30 05:06:31.688938 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:31.688884    5898 scope.go:122] "RemoveContainer" containerID="d986aaa1c60a4e8b83b62a97ac81d10dbe97c96b9bc26b4574d3687156115840"
Oct 30 05:06:31.736896 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:31.736867    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:34.099265 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:34.099221    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-c613f5ea/e2e-lifecycle-model-c613f5ea-kserve-5bf54d4c45-n5286"
Oct 30 05:06:34.116773 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:34.116744    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-c613f5ea/e2e-lifecycle-model-c613f5ea-kserve-5bf54d4c45-n5286"
Oct 30 05:06:34.163619 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:34.163586    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:34.280500 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:34.280457    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:34.350112 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:34.350036    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-c613f5ea/e2e-lifecycle-model-c613f5ea-kserve-5bf54d4c45-n5286"
Oct 30 05:06:34.822205 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:34.822172    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:35.382949 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:35.382892    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:35.591408 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:35.591364    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:36.387783 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:36.387747    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:38.208678 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:38.208644    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:39.251262 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:39.251218    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:39.369994 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:39.369962    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:39.742544 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:39.742506    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:40.240777 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:40.240746    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:40.997839 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:40.997793    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:41.104639 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:41.104604    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:41.762351 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:41.762316    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:42.238040 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:42.238000    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:42.386151 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:42.386099    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:42.511527 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:42.511439    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:45.770372 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:45.770344    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:46.114676 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:46.114643    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:46.257114 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:46.257081    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:46.572324 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:46.572296    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:48.017292 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:48.017254    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:48.140425 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:48.140390    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:49.007091 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:49.007057    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:51.191701 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:51.191663    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:51.453436 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:51.453347    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:06:51.700190 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:51.700153    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:52.130034 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:52.129995    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc"]
Oct 30 05:06:52.134766 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:52.134741    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc"
Oct 30 05:06:52.154947 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:52.154921    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-ef844616-openshift-default-dockercfg-2pgmr\""
Oct 30 05:06:52.174663 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:52.174629    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc"]
Oct 30 05:06:52.196147 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:52.196112    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:52.255688 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:52.255658    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eed8971-bba2-4f8d-9e3c-4f95f426dab9-istiod-ca-cert\") pod \"e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc\" (UID: \"9eed8971-bba2-4f8d-9e3c-4f95f426dab9\") " pod="openshift-ingress/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc"
Oct 30 05:06:52.255856 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:52.255744    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eed8971-bba2-4f8d-9e3c-4f95f426dab9-credential-socket\") pod \"e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc\" (UID: \"9eed8971-bba2-4f8d-9e3c-4f95f426dab9\") " pod="openshift-ingress/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc"
Oct 30 05:06:52.255856 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:52.255845    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eed8971-bba2-4f8d-9e3c-4f95f426dab9-workload-socket\") pod \"e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc\" (UID: \"9eed8971-bba2-4f8d-9e3c-4f95f426dab9\") " pod="openshift-ingress/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc"
Oct 30 05:06:52.255993 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:52.255975    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eed8971-bba2-4f8d-9e3c-4f95f426dab9-istio-data\") pod \"e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc\" (UID: \"9eed8971-bba2-4f8d-9e3c-4f95f426dab9\") " pod="openshift-ingress/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc"
Oct 30 05:06:52.256032 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:52.256013    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eed8971-bba2-4f8d-9e3c-4f95f426dab9-istio-podinfo\") pod \"e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc\" (UID: \"9eed8971-bba2-4f8d-9e3c-4f95f426dab9\") " pod="openshift-ingress/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc"
Oct 30 05:06:52.256071 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:52.256039    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eed8971-bba2-4f8d-9e3c-4f95f426dab9-workload-certs\") pod \"e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc\" (UID: \"9eed8971-bba2-4f8d-9e3c-4f95f426dab9\") " pod="openshift-ingress/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc"
Oct 30 05:06:52.256187 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:52.256162    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eed8971-bba2-4f8d-9e3c-4f95f426dab9-istio-envoy\") pod \"e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc\" (UID: \"9eed8971-bba2-4f8d-9e3c-4f95f426dab9\") " pod="openshift-ingress/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc"
Oct 30 05:06:52.256238 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:52.256225    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp5pc\" (UniqueName: \"kubernetes.io/projected/9eed8971-bba2-4f8d-9e3c-4f95f426dab9-kube-api-access-jp5pc\") pod \"e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc\" (UID: \"9eed8971-bba2-4f8d-9e3c-4f95f426dab9\") " pod="openshift-ingress/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc"
Oct 30 05:06:52.256290 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:52.256278    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eed8971-bba2-4f8d-9e3c-4f95f426dab9-istio-token\") pod \"e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc\" (UID: \"9eed8971-bba2-4f8d-9e3c-4f95f426dab9\") " pod="openshift-ingress/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc"
Oct 30 05:06:52.317697 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:52.317664    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:52.358313 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:52.358266    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eed8971-bba2-4f8d-9e3c-4f95f426dab9-istio-data\") pod \"e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc\" (UID: \"9eed8971-bba2-4f8d-9e3c-4f95f426dab9\") " pod="openshift-ingress/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc"
Oct 30 05:06:52.358609 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:52.358575    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eed8971-bba2-4f8d-9e3c-4f95f426dab9-istio-podinfo\") pod \"e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc\" (UID: \"9eed8971-bba2-4f8d-9e3c-4f95f426dab9\") " pod="openshift-ingress/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc"
Oct 30 05:06:52.358763 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:52.358747    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eed8971-bba2-4f8d-9e3c-4f95f426dab9-workload-certs\") pod \"e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc\" (UID: \"9eed8971-bba2-4f8d-9e3c-4f95f426dab9\") " pod="openshift-ingress/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc"
Oct 30 05:06:52.358941 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:52.358925    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eed8971-bba2-4f8d-9e3c-4f95f426dab9-istio-envoy\") pod \"e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc\" (UID: \"9eed8971-bba2-4f8d-9e3c-4f95f426dab9\") " pod="openshift-ingress/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc"
Oct 30 05:06:52.359138 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:52.359119    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp5pc\" (UniqueName: \"kubernetes.io/projected/9eed8971-bba2-4f8d-9e3c-4f95f426dab9-kube-api-access-jp5pc\") pod \"e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc\" (UID: \"9eed8971-bba2-4f8d-9e3c-4f95f426dab9\") " pod="openshift-ingress/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc"
Oct 30 05:06:52.359334 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:52.359319    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eed8971-bba2-4f8d-9e3c-4f95f426dab9-istio-token\") pod \"e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc\" (UID: \"9eed8971-bba2-4f8d-9e3c-4f95f426dab9\") " pod="openshift-ingress/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc"
Oct 30 05:06:52.359517 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:52.359485    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eed8971-bba2-4f8d-9e3c-4f95f426dab9-istiod-ca-cert\") pod \"e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc\" (UID: \"9eed8971-bba2-4f8d-9e3c-4f95f426dab9\") " pod="openshift-ingress/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc"
Oct 30 05:06:52.359755 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:52.359736    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eed8971-bba2-4f8d-9e3c-4f95f426dab9-credential-socket\") pod \"e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc\" (UID: \"9eed8971-bba2-4f8d-9e3c-4f95f426dab9\") " pod="openshift-ingress/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc"
Oct 30 05:06:52.360076 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:52.360046    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eed8971-bba2-4f8d-9e3c-4f95f426dab9-workload-socket\") pod \"e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc\" (UID: \"9eed8971-bba2-4f8d-9e3c-4f95f426dab9\") " pod="openshift-ingress/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc"
Oct 30 05:06:52.376517 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:52.376486    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eed8971-bba2-4f8d-9e3c-4f95f426dab9-workload-socket\") pod \"e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc\" (UID: \"9eed8971-bba2-4f8d-9e3c-4f95f426dab9\") " pod="openshift-ingress/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc"
Oct 30 05:06:52.380118 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:52.378164    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eed8971-bba2-4f8d-9e3c-4f95f426dab9-istiod-ca-cert\") pod \"e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc\" (UID: \"9eed8971-bba2-4f8d-9e3c-4f95f426dab9\") " pod="openshift-ingress/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc"
Oct 30 05:06:52.380118 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:52.378730    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eed8971-bba2-4f8d-9e3c-4f95f426dab9-istio-data\") pod \"e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc\" (UID: \"9eed8971-bba2-4f8d-9e3c-4f95f426dab9\") " pod="openshift-ingress/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc"
Oct 30 05:06:52.380118 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:52.379026    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eed8971-bba2-4f8d-9e3c-4f95f426dab9-credential-socket\") pod \"e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc\" (UID: \"9eed8971-bba2-4f8d-9e3c-4f95f426dab9\") " pod="openshift-ingress/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc"
Oct 30 05:06:52.380118 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:52.379269    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eed8971-bba2-4f8d-9e3c-4f95f426dab9-workload-certs\") pod \"e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc\" (UID: \"9eed8971-bba2-4f8d-9e3c-4f95f426dab9\") " pod="openshift-ingress/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc"
Oct 30 05:06:52.425925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:52.420660    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eed8971-bba2-4f8d-9e3c-4f95f426dab9-istio-token\") pod \"e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc\" (UID: \"9eed8971-bba2-4f8d-9e3c-4f95f426dab9\") " pod="openshift-ingress/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc"
Oct 30 05:06:52.425925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:52.421353    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp5pc\" (UniqueName: \"kubernetes.io/projected/9eed8971-bba2-4f8d-9e3c-4f95f426dab9-kube-api-access-jp5pc\") pod \"e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc\" (UID: \"9eed8971-bba2-4f8d-9e3c-4f95f426dab9\") " pod="openshift-ingress/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc"
Oct 30 05:06:52.458922 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:52.445528    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eed8971-bba2-4f8d-9e3c-4f95f426dab9-istio-podinfo\") pod \"e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc\" (UID: \"9eed8971-bba2-4f8d-9e3c-4f95f426dab9\") " pod="openshift-ingress/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc"
Oct 30 05:06:52.458922 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:52.445722    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eed8971-bba2-4f8d-9e3c-4f95f426dab9-istio-envoy\") pod \"e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc\" (UID: \"9eed8971-bba2-4f8d-9e3c-4f95f426dab9\") " pod="openshift-ingress/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc"
Oct 30 05:06:52.458922 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:52.454162    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc"
Oct 30 05:06:53.015197 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:53.015157    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc"]
Oct 30 05:06:53.030999 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:53.030952    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:06:53.031156 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:53.031051    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:06:53.031156 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:53.031098    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:06:53.625503 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:53.625459    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc" event={"ID":"9eed8971-bba2-4f8d-9e3c-4f95f426dab9","Type":"ContainerStarted","Data":"dbf57086b26e517fa9da3dd64a6356e1d905c90726702ec0287a77262d56c01b"}
Oct 30 05:06:53.626063 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:53.625534    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc" event={"ID":"9eed8971-bba2-4f8d-9e3c-4f95f426dab9","Type":"ContainerStarted","Data":"2933edd40042378c0b5ed6fa9c8afec2f1e807cdb6fa026851a597c41ee41d24"}
Oct 30 05:06:53.698444 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:53.698383    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc" podStartSLOduration=1.69836472 podStartE2EDuration="1.69836472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:06:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:06:53.680595251 +0000 UTC m=+1342.904415151" watchObservedRunningTime="2026-10-30 05:06:53.69836472 +0000 UTC m=+1342.922184624"
Oct 30 05:06:53.769535 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:53.769501    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:54.455967 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:54.455930    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc"
Oct 30 05:06:54.461650 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:54.461623    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc"
Oct 30 05:06:54.637642 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:54.637599    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc"
Oct 30 05:06:54.639146 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:54.639124    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc"
Oct 30 05:06:54.702684 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:54.700141    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:55.188217 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:55.188187    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:55.632300 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:55.632259    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:55.732700 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:55.732668    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:56.233216 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:56.233180    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:56.384296 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:56.384256    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:56.586920 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:56.583641    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:56.820926 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:06:56.820142    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:06:57.986807 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:57.986775    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:58.186194 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:58.186154    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:58.391025 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:58.390993    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:59.698451 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:59.698417    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:59.842763 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:59.842730    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:06:59.999234 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:06:59.999147    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:07:00.072376 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:00.072344    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5"]
Oct 30 05:07:00.077230 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:00.077207    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5"
Oct 30 05:07:00.092448 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:00.092418    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-ac96f3-openshift-default-dockercfg-ppqp4\""
Oct 30 05:07:00.131124 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:00.126036    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5"]
Oct 30 05:07:00.135167 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:00.135125    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:07:00.184539 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:00.184507    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ec75360-8dfb-45e6-85cc-267e154966f8-istiod-ca-cert\") pod \"e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5\" (UID: \"0ec75360-8dfb-45e6-85cc-267e154966f8\") " pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5"
Oct 30 05:07:00.184750 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:00.184722    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ec75360-8dfb-45e6-85cc-267e154966f8-istio-podinfo\") pod \"e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5\" (UID: \"0ec75360-8dfb-45e6-85cc-267e154966f8\") " pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5"
Oct 30 05:07:00.184797 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:00.184775    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ec75360-8dfb-45e6-85cc-267e154966f8-istio-token\") pod \"e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5\" (UID: \"0ec75360-8dfb-45e6-85cc-267e154966f8\") " pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5"
Oct 30 05:07:00.184864 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:00.184844    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ec75360-8dfb-45e6-85cc-267e154966f8-istio-data\") pod \"e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5\" (UID: \"0ec75360-8dfb-45e6-85cc-267e154966f8\") " pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5"
Oct 30 05:07:00.184948 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:00.184896    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ec75360-8dfb-45e6-85cc-267e154966f8-workload-certs\") pod \"e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5\" (UID: \"0ec75360-8dfb-45e6-85cc-267e154966f8\") " pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5"
Oct 30 05:07:00.185050 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:00.185031    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z42ch\" (UniqueName: \"kubernetes.io/projected/0ec75360-8dfb-45e6-85cc-267e154966f8-kube-api-access-z42ch\") pod \"e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5\" (UID: \"0ec75360-8dfb-45e6-85cc-267e154966f8\") " pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5"
Oct 30 05:07:00.185090 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:00.185071    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ec75360-8dfb-45e6-85cc-267e154966f8-credential-socket\") pod \"e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5\" (UID: \"0ec75360-8dfb-45e6-85cc-267e154966f8\") " pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5"
Oct 30 05:07:00.185269 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:00.185253    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ec75360-8dfb-45e6-85cc-267e154966f8-workload-socket\") pod \"e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5\" (UID: \"0ec75360-8dfb-45e6-85cc-267e154966f8\") " pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5"
Oct 30 05:07:00.185324 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:00.185310    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ec75360-8dfb-45e6-85cc-267e154966f8-istio-envoy\") pod \"e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5\" (UID: \"0ec75360-8dfb-45e6-85cc-267e154966f8\") " pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5"
Oct 30 05:07:00.284999 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:00.275382    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:07:00.296922 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:00.290042    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ec75360-8dfb-45e6-85cc-267e154966f8-istio-podinfo\") pod \"e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5\" (UID: \"0ec75360-8dfb-45e6-85cc-267e154966f8\") " pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5"
Oct 30 05:07:00.296922 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:00.290107    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ec75360-8dfb-45e6-85cc-267e154966f8-istio-token\") pod \"e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5\" (UID: \"0ec75360-8dfb-45e6-85cc-267e154966f8\") " pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5"
Oct 30 05:07:00.296922 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:00.290146    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ec75360-8dfb-45e6-85cc-267e154966f8-istio-data\") pod \"e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5\" (UID: \"0ec75360-8dfb-45e6-85cc-267e154966f8\") " pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5"
Oct 30 05:07:00.296922 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:00.290206    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ec75360-8dfb-45e6-85cc-267e154966f8-workload-certs\") pod \"e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5\" (UID: \"0ec75360-8dfb-45e6-85cc-267e154966f8\") " pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5"
Oct 30 05:07:00.296922 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:00.290257    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z42ch\" (UniqueName: \"kubernetes.io/projected/0ec75360-8dfb-45e6-85cc-267e154966f8-kube-api-access-z42ch\") pod \"e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5\" (UID: \"0ec75360-8dfb-45e6-85cc-267e154966f8\") " pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5"
Oct 30 05:07:00.296922 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:00.290297    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ec75360-8dfb-45e6-85cc-267e154966f8-credential-socket\") pod \"e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5\" (UID: \"0ec75360-8dfb-45e6-85cc-267e154966f8\") " pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5"
Oct 30 05:07:00.296922 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:00.290464    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ec75360-8dfb-45e6-85cc-267e154966f8-workload-socket\") pod \"e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5\" (UID: \"0ec75360-8dfb-45e6-85cc-267e154966f8\") " pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5"
Oct 30 05:07:00.296922 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:00.290525    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ec75360-8dfb-45e6-85cc-267e154966f8-istio-envoy\") pod \"e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5\" (UID: \"0ec75360-8dfb-45e6-85cc-267e154966f8\") " pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5"
Oct 30 05:07:00.296922 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:00.290595    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ec75360-8dfb-45e6-85cc-267e154966f8-istiod-ca-cert\") pod \"e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5\" (UID: \"0ec75360-8dfb-45e6-85cc-267e154966f8\") " pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5"
Oct 30 05:07:00.379952 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:00.370040    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ec75360-8dfb-45e6-85cc-267e154966f8-istiod-ca-cert\") pod \"e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5\" (UID: \"0ec75360-8dfb-45e6-85cc-267e154966f8\") " pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5"
Oct 30 05:07:00.379952 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:00.370387    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ec75360-8dfb-45e6-85cc-267e154966f8-credential-socket\") pod \"e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5\" (UID: \"0ec75360-8dfb-45e6-85cc-267e154966f8\") " pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5"
Oct 30 05:07:00.379952 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:00.374275    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ec75360-8dfb-45e6-85cc-267e154966f8-workload-socket\") pod \"e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5\" (UID: \"0ec75360-8dfb-45e6-85cc-267e154966f8\") " pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5"
Oct 30 05:07:00.405913 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:00.389643    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ec75360-8dfb-45e6-85cc-267e154966f8-istio-data\") pod \"e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5\" (UID: \"0ec75360-8dfb-45e6-85cc-267e154966f8\") " pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5"
Oct 30 05:07:00.405913 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:00.390401    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ec75360-8dfb-45e6-85cc-267e154966f8-workload-certs\") pod \"e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5\" (UID: \"0ec75360-8dfb-45e6-85cc-267e154966f8\") " pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5"
Oct 30 05:07:00.410787 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:00.410757    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ec75360-8dfb-45e6-85cc-267e154966f8-istio-envoy\") pod \"e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5\" (UID: \"0ec75360-8dfb-45e6-85cc-267e154966f8\") " pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5"
Oct 30 05:07:00.455761 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:00.416537    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ec75360-8dfb-45e6-85cc-267e154966f8-istio-token\") pod \"e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5\" (UID: \"0ec75360-8dfb-45e6-85cc-267e154966f8\") " pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5"
Oct 30 05:07:00.456402 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:00.456321    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ec75360-8dfb-45e6-85cc-267e154966f8-istio-podinfo\") pod \"e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5\" (UID: \"0ec75360-8dfb-45e6-85cc-267e154966f8\") " pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5"
Oct 30 05:07:00.461884 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:00.459634    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z42ch\" (UniqueName: \"kubernetes.io/projected/0ec75360-8dfb-45e6-85cc-267e154966f8-kube-api-access-z42ch\") pod \"e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5\" (UID: \"0ec75360-8dfb-45e6-85cc-267e154966f8\") " pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5"
Oct 30 05:07:00.649958 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:00.649487    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:07:00.697236 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:00.697200    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5"
Oct 30 05:07:01.037333 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:07:01.037295    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ec75360_8dfb_45e6_85cc_267e154966f8.slice/crio-75a637b30f6e1b11c80765f374f4e1c0c7dc1406dd50a1323497387c77bee2db WatchSource:0}: Error finding container 75a637b30f6e1b11c80765f374f4e1c0c7dc1406dd50a1323497387c77bee2db: Status 404 returned error can't find the container with id 75a637b30f6e1b11c80765f374f4e1c0c7dc1406dd50a1323497387c77bee2db
Oct 30 05:07:01.040071 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:01.040037    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:07:01.040165 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:01.040122    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:07:01.040204 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:01.040162    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:07:01.040299 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:01.040275    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5"]
Oct 30 05:07:01.644998 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:01.644967    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:07:01.719935 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:01.719850    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5" event={"ID":"0ec75360-8dfb-45e6-85cc-267e154966f8","Type":"ContainerStarted","Data":"dde1623074ec63842dffea36914a2b3fc9704c56c5df4478d23b263b1b8c50f8"}
Oct 30 05:07:01.719935 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:01.719924    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5" event={"ID":"0ec75360-8dfb-45e6-85cc-267e154966f8","Type":"ContainerStarted","Data":"75a637b30f6e1b11c80765f374f4e1c0c7dc1406dd50a1323497387c77bee2db"}
Oct 30 05:07:01.776462 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:01.776399    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5" podStartSLOduration=1.776379358 podStartE2EDuration="1.776379358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:07:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:07:01.771818672 +0000 UTC m=+1350.995638570" watchObservedRunningTime="2026-10-30 05:07:01.776379358 +0000 UTC m=+1351.000199260"
Oct 30 05:07:01.778338 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:01.778302    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:07:02.697760 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:02.697726    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5"
Oct 30 05:07:02.703708 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:02.703683    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5"
Oct 30 05:07:02.705603 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:02.705578    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:07:02.730954 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:02.729753    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5"
Oct 30 05:07:02.730954 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:02.730693    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5"
Oct 30 05:07:03.029294 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:03.029202    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-566b2bc6-hbgtv"]
Oct 30 05:07:03.033217 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:03.033189    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-566b2bc6-hbgtv"]
Oct 30 05:07:04.164910 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:04.164862    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:07:04.276105 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:04.276075    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:07:04.420240 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:04.420158    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k"]
Oct 30 05:07:04.425121 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:04.425084    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k"
Oct 30 05:07:04.441687 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:04.441665    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-c613f5ea-openshift-default-dockercfg-dxwcw\""
Oct 30 05:07:04.458307 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:04.458278    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k"]
Oct 30 05:07:04.477601 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:04.477564    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/536d83f7-b214-4e92-ad4d-5b00c0fef6aa-istio-podinfo\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k\" (UID: \"536d83f7-b214-4e92-ad4d-5b00c0fef6aa\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k"
Oct 30 05:07:04.477765 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:04.477739    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/536d83f7-b214-4e92-ad4d-5b00c0fef6aa-istio-token\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k\" (UID: \"536d83f7-b214-4e92-ad4d-5b00c0fef6aa\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k"
Oct 30 05:07:04.477827 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:04.477803    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/536d83f7-b214-4e92-ad4d-5b00c0fef6aa-istiod-ca-cert\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k\" (UID: \"536d83f7-b214-4e92-ad4d-5b00c0fef6aa\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k"
Oct 30 05:07:04.477965 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:04.477947    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/536d83f7-b214-4e92-ad4d-5b00c0fef6aa-workload-socket\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k\" (UID: \"536d83f7-b214-4e92-ad4d-5b00c0fef6aa\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k"
Oct 30 05:07:04.478022 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:04.478005    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/536d83f7-b214-4e92-ad4d-5b00c0fef6aa-workload-certs\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k\" (UID: \"536d83f7-b214-4e92-ad4d-5b00c0fef6aa\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k"
Oct 30 05:07:04.478133 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:04.478116    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6hgk\" (UniqueName: \"kubernetes.io/projected/536d83f7-b214-4e92-ad4d-5b00c0fef6aa-kube-api-access-h6hgk\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k\" (UID: \"536d83f7-b214-4e92-ad4d-5b00c0fef6aa\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k"
Oct 30 05:07:04.478181 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:04.478158    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/536d83f7-b214-4e92-ad4d-5b00c0fef6aa-credential-socket\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k\" (UID: \"536d83f7-b214-4e92-ad4d-5b00c0fef6aa\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k"
Oct 30 05:07:04.478241 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:04.478225    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/536d83f7-b214-4e92-ad4d-5b00c0fef6aa-istio-data\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k\" (UID: \"536d83f7-b214-4e92-ad4d-5b00c0fef6aa\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k"
Oct 30 05:07:04.478299 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:04.478285    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/536d83f7-b214-4e92-ad4d-5b00c0fef6aa-istio-envoy\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k\" (UID: \"536d83f7-b214-4e92-ad4d-5b00c0fef6aa\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k"
Oct 30 05:07:04.580228 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:04.580196    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6hgk\" (UniqueName: \"kubernetes.io/projected/536d83f7-b214-4e92-ad4d-5b00c0fef6aa-kube-api-access-h6hgk\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k\" (UID: \"536d83f7-b214-4e92-ad4d-5b00c0fef6aa\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k"
Oct 30 05:07:04.580383 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:04.580239    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/536d83f7-b214-4e92-ad4d-5b00c0fef6aa-credential-socket\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k\" (UID: \"536d83f7-b214-4e92-ad4d-5b00c0fef6aa\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k"
Oct 30 05:07:04.580383 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:04.580279    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/536d83f7-b214-4e92-ad4d-5b00c0fef6aa-istio-data\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k\" (UID: \"536d83f7-b214-4e92-ad4d-5b00c0fef6aa\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k"
Oct 30 05:07:04.580383 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:04.580312    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/536d83f7-b214-4e92-ad4d-5b00c0fef6aa-istio-envoy\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k\" (UID: \"536d83f7-b214-4e92-ad4d-5b00c0fef6aa\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k"
Oct 30 05:07:04.580569 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:04.580380    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/536d83f7-b214-4e92-ad4d-5b00c0fef6aa-istio-podinfo\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k\" (UID: \"536d83f7-b214-4e92-ad4d-5b00c0fef6aa\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k"
Oct 30 05:07:04.580569 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:04.580472    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/536d83f7-b214-4e92-ad4d-5b00c0fef6aa-istio-token\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k\" (UID: \"536d83f7-b214-4e92-ad4d-5b00c0fef6aa\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k"
Oct 30 05:07:04.580569 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:04.580519    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/536d83f7-b214-4e92-ad4d-5b00c0fef6aa-istiod-ca-cert\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k\" (UID: \"536d83f7-b214-4e92-ad4d-5b00c0fef6aa\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k"
Oct 30 05:07:04.580728 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:04.580619    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/536d83f7-b214-4e92-ad4d-5b00c0fef6aa-workload-socket\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k\" (UID: \"536d83f7-b214-4e92-ad4d-5b00c0fef6aa\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k"
Oct 30 05:07:04.580728 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:04.580671    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/536d83f7-b214-4e92-ad4d-5b00c0fef6aa-workload-certs\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k\" (UID: \"536d83f7-b214-4e92-ad4d-5b00c0fef6aa\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k"
Oct 30 05:07:04.580835 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:04.580738    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/536d83f7-b214-4e92-ad4d-5b00c0fef6aa-istio-data\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k\" (UID: \"536d83f7-b214-4e92-ad4d-5b00c0fef6aa\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k"
Oct 30 05:07:04.580835 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:04.580808    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/536d83f7-b214-4e92-ad4d-5b00c0fef6aa-credential-socket\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k\" (UID: \"536d83f7-b214-4e92-ad4d-5b00c0fef6aa\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k"
Oct 30 05:07:04.581052 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:04.581030    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/536d83f7-b214-4e92-ad4d-5b00c0fef6aa-workload-socket\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k\" (UID: \"536d83f7-b214-4e92-ad4d-5b00c0fef6aa\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k"
Oct 30 05:07:04.581109 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:04.581088    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/536d83f7-b214-4e92-ad4d-5b00c0fef6aa-workload-certs\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k\" (UID: \"536d83f7-b214-4e92-ad4d-5b00c0fef6aa\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k"
Oct 30 05:07:04.581336 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:04.581318    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/536d83f7-b214-4e92-ad4d-5b00c0fef6aa-istiod-ca-cert\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k\" (UID: \"536d83f7-b214-4e92-ad4d-5b00c0fef6aa\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k"
Oct 30 05:07:04.582790 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:04.582764    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/536d83f7-b214-4e92-ad4d-5b00c0fef6aa-istio-envoy\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k\" (UID: \"536d83f7-b214-4e92-ad4d-5b00c0fef6aa\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k"
Oct 30 05:07:04.583040 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:04.583020    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/536d83f7-b214-4e92-ad4d-5b00c0fef6aa-istio-podinfo\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k\" (UID: \"536d83f7-b214-4e92-ad4d-5b00c0fef6aa\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k"
Oct 30 05:07:04.599953 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:04.599922    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6hgk\" (UniqueName: \"kubernetes.io/projected/536d83f7-b214-4e92-ad4d-5b00c0fef6aa-kube-api-access-h6hgk\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k\" (UID: \"536d83f7-b214-4e92-ad4d-5b00c0fef6aa\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k"
Oct 30 05:07:04.600069 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:04.599976    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/536d83f7-b214-4e92-ad4d-5b00c0fef6aa-istio-token\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k\" (UID: \"536d83f7-b214-4e92-ad4d-5b00c0fef6aa\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k"
Oct 30 05:07:04.740645 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:04.740543    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k"
Oct 30 05:07:04.896270 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:04.896231    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:07:04.913623 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:04.913584    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25bc3551-4714-473b-815a-2844fbdcdb59" path="/var/lib/kubelet/pods/25bc3551-4714-473b-815a-2844fbdcdb59/volumes"
Oct 30 05:07:05.061525 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:07:05.061492    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod536d83f7_b214_4e92_ad4d_5b00c0fef6aa.slice/crio-41125c0dfefcbded30abed56f5b156a1e9069f221995139351069ad20ded8369 WatchSource:0}: Error finding container 41125c0dfefcbded30abed56f5b156a1e9069f221995139351069ad20ded8369: Status 404 returned error can't find the container with id 41125c0dfefcbded30abed56f5b156a1e9069f221995139351069ad20ded8369
Oct 30 05:07:05.062407 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:05.062379    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k"]
Oct 30 05:07:05.064434 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:05.064404    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:07:05.064518 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:05.064466    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:07:05.064518 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:05.064495    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:07:05.753423 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:05.753383    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k" event={"ID":"536d83f7-b214-4e92-ad4d-5b00c0fef6aa","Type":"ContainerStarted","Data":"52617c059b3d70542bae1850a8e510c67ead5fb865513c85a52b20466157f671"}
Oct 30 05:07:05.753815 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:05.753448    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k" event={"ID":"536d83f7-b214-4e92-ad4d-5b00c0fef6aa","Type":"ContainerStarted","Data":"41125c0dfefcbded30abed56f5b156a1e9069f221995139351069ad20ded8369"}
Oct 30 05:07:05.813554 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:05.813488    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k" podStartSLOduration=1.8134698 podStartE2EDuration="1.8134698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:07:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:07:05.811759377 +0000 UTC m=+1355.035579279" watchObservedRunningTime="2026-10-30 05:07:05.8134698 +0000 UTC m=+1355.037289704"
Oct 30 05:07:05.920613 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:05.920576    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:07:06.037465 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:06.037368    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:07:06.740978 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:06.740944    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k"
Oct 30 05:07:06.742477 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:06.742456    5898 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 05:07:06.742558 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:06.742502    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k" podUID="536d83f7-b214-4e92-ad4d-5b00c0fef6aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 05:07:06.832389 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:06.832352    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:07:06.837308 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:06.837278    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-93bad9-54ff46887b-kzztl"]
Oct 30 05:07:06.843092 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:06.843068    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-93bad9-54ff46887b-kzztl"
Oct 30 05:07:06.858853 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:06.858826    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-93bad9-serving-cert\""
Oct 30 05:07:06.862713 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:06.862690    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-93bad9-54ff46887b-kzztl"]
Oct 30 05:07:06.931538 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:06.931504    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:07:06.944760 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:06.944716    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-93bad9-54ff46887b-kzztl\" (UID: \"f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-93bad9-54ff46887b-kzztl"
Oct 30 05:07:06.945082 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:06.945056    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-93bad9-54ff46887b-kzztl\" (UID: \"f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-93bad9-54ff46887b-kzztl"
Oct 30 05:07:06.945321 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:06.945301    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4ss2\" (UniqueName: \"kubernetes.io/projected/f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc-kube-api-access-t4ss2\") pod \"maas-api-e2e-shared-b-w2-93bad9-54ff46887b-kzztl\" (UID: \"f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-93bad9-54ff46887b-kzztl"
Oct 30 05:07:06.945484 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:06.945467    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-93bad9-54ff46887b-kzztl\" (UID: \"f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-93bad9-54ff46887b-kzztl"
Oct 30 05:07:07.026672 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:07.026596    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:07:07.047642 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:07.047605    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-93bad9-54ff46887b-kzztl\" (UID: \"f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-93bad9-54ff46887b-kzztl"
Oct 30 05:07:07.047834 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:07.047676    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-93bad9-54ff46887b-kzztl\" (UID: \"f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-93bad9-54ff46887b-kzztl"
Oct 30 05:07:07.047929 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:07.047832    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4ss2\" (UniqueName: \"kubernetes.io/projected/f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc-kube-api-access-t4ss2\") pod \"maas-api-e2e-shared-b-w2-93bad9-54ff46887b-kzztl\" (UID: \"f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-93bad9-54ff46887b-kzztl"
Oct 30 05:07:07.047995 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:07.047929    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-93bad9-54ff46887b-kzztl\" (UID: \"f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-93bad9-54ff46887b-kzztl"
Oct 30 05:07:07.051643 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:07.051615    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-93bad9-54ff46887b-kzztl\" (UID: \"f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-93bad9-54ff46887b-kzztl"
Oct 30 05:07:07.084919 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:07.076893    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4ss2\" (UniqueName: \"kubernetes.io/projected/f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc-kube-api-access-t4ss2\") pod \"maas-api-e2e-shared-b-w2-93bad9-54ff46887b-kzztl\" (UID: \"f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-93bad9-54ff46887b-kzztl"
Oct 30 05:07:07.084919 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:07.077496    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-93bad9-54ff46887b-kzztl\" (UID: \"f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-93bad9-54ff46887b-kzztl"
Oct 30 05:07:07.084919 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:07.081183    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-93bad9-54ff46887b-kzztl\" (UID: \"f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-93bad9-54ff46887b-kzztl"
Oct 30 05:07:07.159159 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:07.159126    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-93bad9-54ff46887b-kzztl"
Oct 30 05:07:07.502843 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:07.502812    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-93bad9-54ff46887b-kzztl"]
Oct 30 05:07:07.504360 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:07:07.504331    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8daa6d5_495f_4b98_abac_6dc2f9b4dbbc.slice/crio-91fea727d1a6abc5ea1ac24e0ec12f2667ccc77e0efe645129173ad5ba29c190 WatchSource:0}: Error finding container 91fea727d1a6abc5ea1ac24e0ec12f2667ccc77e0efe645129173ad5ba29c190: Status 404 returned error can't find the container with id 91fea727d1a6abc5ea1ac24e0ec12f2667ccc77e0efe645129173ad5ba29c190
Oct 30 05:07:07.741737 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:07.741690    5898 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 05:07:07.741988 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:07.741749    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k" podUID="536d83f7-b214-4e92-ad4d-5b00c0fef6aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 05:07:07.772130 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:07.772047    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-93bad9-54ff46887b-kzztl" event={"ID":"f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc","Type":"ContainerStarted","Data":"91fea727d1a6abc5ea1ac24e0ec12f2667ccc77e0efe645129173ad5ba29c190"}
Oct 30 05:07:08.741757 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:08.741724    5898 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 05:07:08.742213 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:08.741775    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k" podUID="536d83f7-b214-4e92-ad4d-5b00c0fef6aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 05:07:08.806600 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:08.806559    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-93bad9-54ff46887b-kzztl" event={"ID":"f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc","Type":"ContainerStarted","Data":"33c6f507394bef4f9461f01a748b74f4477a01a6d8dceb4b9f964a623f006cb7"}
Oct 30 05:07:08.806778 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:08.806639    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-93bad9-54ff46887b-kzztl"
Oct 30 05:07:08.851309 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:08.851258    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-93bad9-54ff46887b-kzztl" podStartSLOduration=2.474247558 podStartE2EDuration="2.851244021s" totalImagesPullingTime="376.996463ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:07:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:07:08.846574698 +0000 UTC m=+1358.070394601" watchObservedRunningTime="2026-10-30 05:07:08.851244021 +0000 UTC m=+1358.075063919"
Oct 30 05:07:09.106889 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:09.106857    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:07:09.742013 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:09.741972    5898 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 05:07:09.742416 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:09.742037    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k" podUID="536d83f7-b214-4e92-ad4d-5b00c0fef6aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 05:07:10.740989 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:10.740947    5898 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 05:07:10.741203 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:10.741001    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k" podUID="536d83f7-b214-4e92-ad4d-5b00c0fef6aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 05:07:10.859399 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:10.859363    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:07:11.741752 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:11.741717    5898 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 05:07:11.741962 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:11.741781    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k" podUID="536d83f7-b214-4e92-ad4d-5b00c0fef6aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 05:07:12.558965 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:12.558885    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:07:12.747121 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:12.747082    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k"
Oct 30 05:07:12.747539 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:12.747516    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k"
Oct 30 05:07:12.748394 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:12.748374    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k"
Oct 30 05:07:14.401774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:14.401735    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:07:14.822335 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:14.822315    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-93bad9-54ff46887b-kzztl"
Oct 30 05:07:16.067056 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:16.067021    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-528d67-bfc8bdc7b-2f26h"]
Oct 30 05:07:16.071648 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:16.071630    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-528d67-bfc8bdc7b-2f26h"
Oct 30 05:07:16.096999 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:16.096964    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-528d67-serving-cert\""
Oct 30 05:07:16.107576 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:16.107541    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-528d67-bfc8bdc7b-2f26h"]
Oct 30 05:07:16.119368 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:16.119332    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1235a781-d216-46c5-b0b2-e39f62ca8c01-metrics-certs\") pod \"maas-api-e2e-api-a-528d67-bfc8bdc7b-2f26h\" (UID: \"1235a781-d216-46c5-b0b2-e39f62ca8c01\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-528d67-bfc8bdc7b-2f26h"
Oct 30 05:07:16.119548 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:16.119494    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1235a781-d216-46c5-b0b2-e39f62ca8c01-openshift-service-ca\") pod \"maas-api-e2e-api-a-528d67-bfc8bdc7b-2f26h\" (UID: \"1235a781-d216-46c5-b0b2-e39f62ca8c01\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-528d67-bfc8bdc7b-2f26h"
Oct 30 05:07:16.119634 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:16.119613    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1235a781-d216-46c5-b0b2-e39f62ca8c01-maas-api-tls\") pod \"maas-api-e2e-api-a-528d67-bfc8bdc7b-2f26h\" (UID: \"1235a781-d216-46c5-b0b2-e39f62ca8c01\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-528d67-bfc8bdc7b-2f26h"
Oct 30 05:07:16.119802 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:16.119778    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpql9\" (UniqueName: \"kubernetes.io/projected/1235a781-d216-46c5-b0b2-e39f62ca8c01-kube-api-access-xpql9\") pod \"maas-api-e2e-api-a-528d67-bfc8bdc7b-2f26h\" (UID: \"1235a781-d216-46c5-b0b2-e39f62ca8c01\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-528d67-bfc8bdc7b-2f26h"
Oct 30 05:07:16.151120 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:16.151081    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:07:16.221079 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:16.221046    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1235a781-d216-46c5-b0b2-e39f62ca8c01-metrics-certs\") pod \"maas-api-e2e-api-a-528d67-bfc8bdc7b-2f26h\" (UID: \"1235a781-d216-46c5-b0b2-e39f62ca8c01\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-528d67-bfc8bdc7b-2f26h"
Oct 30 05:07:16.221279 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:16.221160    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1235a781-d216-46c5-b0b2-e39f62ca8c01-openshift-service-ca\") pod \"maas-api-e2e-api-a-528d67-bfc8bdc7b-2f26h\" (UID: \"1235a781-d216-46c5-b0b2-e39f62ca8c01\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-528d67-bfc8bdc7b-2f26h"
Oct 30 05:07:16.221279 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:16.221241    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1235a781-d216-46c5-b0b2-e39f62ca8c01-maas-api-tls\") pod \"maas-api-e2e-api-a-528d67-bfc8bdc7b-2f26h\" (UID: \"1235a781-d216-46c5-b0b2-e39f62ca8c01\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-528d67-bfc8bdc7b-2f26h"
Oct 30 05:07:16.221458 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:16.221294    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpql9\" (UniqueName: \"kubernetes.io/projected/1235a781-d216-46c5-b0b2-e39f62ca8c01-kube-api-access-xpql9\") pod \"maas-api-e2e-api-a-528d67-bfc8bdc7b-2f26h\" (UID: \"1235a781-d216-46c5-b0b2-e39f62ca8c01\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-528d67-bfc8bdc7b-2f26h"
Oct 30 05:07:16.221975 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:16.221945    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1235a781-d216-46c5-b0b2-e39f62ca8c01-openshift-service-ca\") pod \"maas-api-e2e-api-a-528d67-bfc8bdc7b-2f26h\" (UID: \"1235a781-d216-46c5-b0b2-e39f62ca8c01\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-528d67-bfc8bdc7b-2f26h"
Oct 30 05:07:16.223600 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:16.223576    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1235a781-d216-46c5-b0b2-e39f62ca8c01-metrics-certs\") pod \"maas-api-e2e-api-a-528d67-bfc8bdc7b-2f26h\" (UID: \"1235a781-d216-46c5-b0b2-e39f62ca8c01\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-528d67-bfc8bdc7b-2f26h"
Oct 30 05:07:16.223600 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:16.223592    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1235a781-d216-46c5-b0b2-e39f62ca8c01-maas-api-tls\") pod \"maas-api-e2e-api-a-528d67-bfc8bdc7b-2f26h\" (UID: \"1235a781-d216-46c5-b0b2-e39f62ca8c01\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-528d67-bfc8bdc7b-2f26h"
Oct 30 05:07:16.240655 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:16.240633    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpql9\" (UniqueName: \"kubernetes.io/projected/1235a781-d216-46c5-b0b2-e39f62ca8c01-kube-api-access-xpql9\") pod \"maas-api-e2e-api-a-528d67-bfc8bdc7b-2f26h\" (UID: \"1235a781-d216-46c5-b0b2-e39f62ca8c01\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-528d67-bfc8bdc7b-2f26h"
Oct 30 05:07:16.247662 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:16.247635    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:07:16.350438 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:16.350392    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:07:16.400473 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:16.400170    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-528d67-bfc8bdc7b-2f26h"
Oct 30 05:07:16.729967 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:16.729936    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-528d67-bfc8bdc7b-2f26h"]
Oct 30 05:07:16.731278 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:07:16.731250    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1235a781_d216_46c5_b0b2_e39f62ca8c01.slice/crio-df96a6440d387ed8aefa9b680ada820c99630c0b7ce6bd792dd8fc378bef3c4b WatchSource:0}: Error finding container df96a6440d387ed8aefa9b680ada820c99630c0b7ce6bd792dd8fc378bef3c4b: Status 404 returned error can't find the container with id df96a6440d387ed8aefa9b680ada820c99630c0b7ce6bd792dd8fc378bef3c4b
Oct 30 05:07:16.890231 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:16.890196    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-528d67-bfc8bdc7b-2f26h" event={"ID":"1235a781-d216-46c5-b0b2-e39f62ca8c01","Type":"ContainerStarted","Data":"df96a6440d387ed8aefa9b680ada820c99630c0b7ce6bd792dd8fc378bef3c4b"}
Oct 30 05:07:17.894736 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:17.894695    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-528d67-bfc8bdc7b-2f26h" event={"ID":"1235a781-d216-46c5-b0b2-e39f62ca8c01","Type":"ContainerStarted","Data":"84f0b74943f39ae0127055eb165d90c21eeab526c03835acc75018798d3f50fe"}
Oct 30 05:07:17.895391 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:17.895368    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-528d67-bfc8bdc7b-2f26h"
Oct 30 05:07:17.938067 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:17.937992    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-528d67-bfc8bdc7b-2f26h" podStartSLOduration=1.6379755729999999 podStartE2EDuration="1.937972189s" totalImagesPullingTime="299.996616ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:07:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:07:17.935124973 +0000 UTC m=+1367.158944874" watchObservedRunningTime="2026-10-30 05:07:17.937972189 +0000 UTC m=+1367.161792129"
Oct 30 05:07:18.486787 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:18.486755    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:07:20.353421 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:20.353387    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:07:22.053078 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:22.053040    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:07:22.208683 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:22.208642    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8876-e8f1d5"]
Oct 30 05:07:22.213938 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:22.213914    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8876-e8f1d5"
Oct 30 05:07:22.231516 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:22.231480    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8876-e8f1d5"]
Oct 30 05:07:22.233103 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:22.233079    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-twdr8\""
Oct 30 05:07:22.331357 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:22.331323    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf8b6\" (UniqueName: \"kubernetes.io/projected/5cfa3691-14ae-46c8-9b12-5a6c34dd1562-kube-api-access-wf8b6\") pod \"test-curl-8876-e8f1d5\" (UID: \"5cfa3691-14ae-46c8-9b12-5a6c34dd1562\") " pod="opendatahub/test-curl-8876-e8f1d5"
Oct 30 05:07:22.433432 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:22.433403    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf8b6\" (UniqueName: \"kubernetes.io/projected/5cfa3691-14ae-46c8-9b12-5a6c34dd1562-kube-api-access-wf8b6\") pod \"test-curl-8876-e8f1d5\" (UID: \"5cfa3691-14ae-46c8-9b12-5a6c34dd1562\") " pod="opendatahub/test-curl-8876-e8f1d5"
Oct 30 05:07:22.453462 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:22.453429    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf8b6\" (UniqueName: \"kubernetes.io/projected/5cfa3691-14ae-46c8-9b12-5a6c34dd1562-kube-api-access-wf8b6\") pod \"test-curl-8876-e8f1d5\" (UID: \"5cfa3691-14ae-46c8-9b12-5a6c34dd1562\") " pod="opendatahub/test-curl-8876-e8f1d5"
Oct 30 05:07:22.542250 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:22.542210    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8876-e8f1d5"
Oct 30 05:07:22.842887 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:07:22.842856    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5cfa3691_14ae_46c8_9b12_5a6c34dd1562.slice/crio-0777316c53a637973d63408c208511c5d3c52663840498c3909406a89e125d5e WatchSource:0}: Error finding container 0777316c53a637973d63408c208511c5d3c52663840498c3909406a89e125d5e: Status 404 returned error can't find the container with id 0777316c53a637973d63408c208511c5d3c52663840498c3909406a89e125d5e
Oct 30 05:07:22.843106 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:22.843084    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8876-e8f1d5"]
Oct 30 05:07:22.933633 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:22.933603    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8876-e8f1d5" event={"ID":"5cfa3691-14ae-46c8-9b12-5a6c34dd1562","Type":"ContainerStarted","Data":"8177ce25451a61f708b186374ab576934c14f9ebf1819c946ff94fa120afb02f"}
Oct 30 05:07:22.933752 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:22.933649    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8876-e8f1d5" event={"ID":"5cfa3691-14ae-46c8-9b12-5a6c34dd1562","Type":"ContainerStarted","Data":"0777316c53a637973d63408c208511c5d3c52663840498c3909406a89e125d5e"}
Oct 30 05:07:22.970420 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:22.970333    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8876-e8f1d5" podStartSLOduration=0.970314699 podStartE2EDuration="970.314699ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:07:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:07:22.968353337 +0000 UTC m=+1372.192173237" watchObservedRunningTime="2026-10-30 05:07:22.970314699 +0000 UTC m=+1372.194134597"
Oct 30 05:07:23.813111 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:23.813022    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8876-e8f1d5"]
Oct 30 05:07:23.819821 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:23.819782    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8876-e8f1d5"]
Oct 30 05:07:23.826447 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:23.826411    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:07:23.911665 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:23.909965    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-528d67-bfc8bdc7b-2f26h"
Oct 30 05:07:24.947676 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:24.947613    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8876-e8f1d5" podUID="5cfa3691-14ae-46c8-9b12-5a6c34dd1562" containerName="test-curl-8876-e8f1d5" containerID="cri-o://8177ce25451a61f708b186374ab576934c14f9ebf1819c946ff94fa120afb02f" gracePeriod=2
Oct 30 05:07:25.031421 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:25.031389    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46a9f75e-wbj7s"]
Oct 30 05:07:25.042202 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:25.042178    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46a9f75e-wbj7s"]
Oct 30 05:07:25.537555 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:25.537525    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ef844616-6b67dd9559-bb6pp"]
Oct 30 05:07:25.538281 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:25.538263    5898 cpu_manager.go:414] "containerMap: removing container" podUID="5cfa3691-14ae-46c8-9b12-5a6c34dd1562" containerName="test-curl-8876-e8f1d5"
Oct 30 05:07:25.538281 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:25.538281    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cfa3691-14ae-46c8-9b12-5a6c34dd1562" containerName="test-curl-8876-e8f1d5"
Oct 30 05:07:25.538389 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:25.538384    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cfa3691-14ae-46c8-9b12-5a6c34dd1562" containerName="test-curl-8876-e8f1d5"
Oct 30 05:07:25.542413 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:25.542393    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef844616-6b67dd9559-bb6pp"
Oct 30 05:07:25.560957 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:25.560930    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ef844616-6b67dd9559-bb6pp"]
Oct 30 05:07:25.566701 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:25.566683    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-ef844616-serving-cert\""
Oct 30 05:07:25.614169 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:25.614135    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:07:25.682074 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:25.682037    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5036975c-76af-4820-88b6-3c1b53400520-metrics-certs\") pod \"maas-api-e2e-mt-ef844616-6b67dd9559-bb6pp\" (UID: \"5036975c-76af-4820-88b6-3c1b53400520\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef844616-6b67dd9559-bb6pp"
Oct 30 05:07:25.682239 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:25.682090    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5036975c-76af-4820-88b6-3c1b53400520-maas-api-tls\") pod \"maas-api-e2e-mt-ef844616-6b67dd9559-bb6pp\" (UID: \"5036975c-76af-4820-88b6-3c1b53400520\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef844616-6b67dd9559-bb6pp"
Oct 30 05:07:25.682287 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:25.682248    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2974\" (UniqueName: \"kubernetes.io/projected/5036975c-76af-4820-88b6-3c1b53400520-kube-api-access-g2974\") pod \"maas-api-e2e-mt-ef844616-6b67dd9559-bb6pp\" (UID: \"5036975c-76af-4820-88b6-3c1b53400520\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef844616-6b67dd9559-bb6pp"
Oct 30 05:07:25.682405 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:25.682391    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5036975c-76af-4820-88b6-3c1b53400520-openshift-service-ca\") pod \"maas-api-e2e-mt-ef844616-6b67dd9559-bb6pp\" (UID: \"5036975c-76af-4820-88b6-3c1b53400520\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef844616-6b67dd9559-bb6pp"
Oct 30 05:07:25.714529 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:25.714499    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:07:25.783645 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:25.783609    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5036975c-76af-4820-88b6-3c1b53400520-openshift-service-ca\") pod \"maas-api-e2e-mt-ef844616-6b67dd9559-bb6pp\" (UID: \"5036975c-76af-4820-88b6-3c1b53400520\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef844616-6b67dd9559-bb6pp"
Oct 30 05:07:25.783857 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:25.783754    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5036975c-76af-4820-88b6-3c1b53400520-metrics-certs\") pod \"maas-api-e2e-mt-ef844616-6b67dd9559-bb6pp\" (UID: \"5036975c-76af-4820-88b6-3c1b53400520\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef844616-6b67dd9559-bb6pp"
Oct 30 05:07:25.783857 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:25.783784    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5036975c-76af-4820-88b6-3c1b53400520-maas-api-tls\") pod \"maas-api-e2e-mt-ef844616-6b67dd9559-bb6pp\" (UID: \"5036975c-76af-4820-88b6-3c1b53400520\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef844616-6b67dd9559-bb6pp"
Oct 30 05:07:25.784014 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:25.783870    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2974\" (UniqueName: \"kubernetes.io/projected/5036975c-76af-4820-88b6-3c1b53400520-kube-api-access-g2974\") pod \"maas-api-e2e-mt-ef844616-6b67dd9559-bb6pp\" (UID: \"5036975c-76af-4820-88b6-3c1b53400520\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef844616-6b67dd9559-bb6pp"
Oct 30 05:07:25.784399 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:25.784373    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5036975c-76af-4820-88b6-3c1b53400520-openshift-service-ca\") pod \"maas-api-e2e-mt-ef844616-6b67dd9559-bb6pp\" (UID: \"5036975c-76af-4820-88b6-3c1b53400520\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef844616-6b67dd9559-bb6pp"
Oct 30 05:07:25.786285 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:25.786261    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5036975c-76af-4820-88b6-3c1b53400520-metrics-certs\") pod \"maas-api-e2e-mt-ef844616-6b67dd9559-bb6pp\" (UID: \"5036975c-76af-4820-88b6-3c1b53400520\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef844616-6b67dd9559-bb6pp"
Oct 30 05:07:25.786772 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:25.786536    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5036975c-76af-4820-88b6-3c1b53400520-maas-api-tls\") pod \"maas-api-e2e-mt-ef844616-6b67dd9559-bb6pp\" (UID: \"5036975c-76af-4820-88b6-3c1b53400520\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef844616-6b67dd9559-bb6pp"
Oct 30 05:07:25.804372 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:25.804296    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2974\" (UniqueName: \"kubernetes.io/projected/5036975c-76af-4820-88b6-3c1b53400520-kube-api-access-g2974\") pod \"maas-api-e2e-mt-ef844616-6b67dd9559-bb6pp\" (UID: \"5036975c-76af-4820-88b6-3c1b53400520\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef844616-6b67dd9559-bb6pp"
Oct 30 05:07:25.857198 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:25.857165    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef844616-6b67dd9559-bb6pp"
Oct 30 05:07:26.167079 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:07:26.167033    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5036975c_76af_4820_88b6_3c1b53400520.slice/crio-ae9bdf0574eda35173ab1b51f06e8edc60bf5f823f14a9b00c1cd7f942cf790f WatchSource:0}: Error finding container ae9bdf0574eda35173ab1b51f06e8edc60bf5f823f14a9b00c1cd7f942cf790f: Status 404 returned error can't find the container with id ae9bdf0574eda35173ab1b51f06e8edc60bf5f823f14a9b00c1cd7f942cf790f
Oct 30 05:07:26.168088 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:26.168060    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ef844616-6b67dd9559-bb6pp"]
Oct 30 05:07:26.821529 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:07:26.821486    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:07:26.891792 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:26.891763    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="161f48e4-7e83-44ff-a12b-4f77d53dc5a8" path="/var/lib/kubelet/pods/161f48e4-7e83-44ff-a12b-4f77d53dc5a8/volumes"
Oct 30 05:07:26.979205 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:26.979168    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef844616-6b67dd9559-bb6pp" event={"ID":"5036975c-76af-4820-88b6-3c1b53400520","Type":"ContainerStarted","Data":"e953fab6ce022d52057c484db33d11419c52d25a53ceb6f53d7b2627ae53449b"}
Oct 30 05:07:26.979330 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:26.979238    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef844616-6b67dd9559-bb6pp" event={"ID":"5036975c-76af-4820-88b6-3c1b53400520","Type":"ContainerStarted","Data":"ae9bdf0574eda35173ab1b51f06e8edc60bf5f823f14a9b00c1cd7f942cf790f"}
Oct 30 05:07:26.980485 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:26.980459    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef844616-6b67dd9559-bb6pp"
Oct 30 05:07:27.041328 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:27.041258    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef844616-6b67dd9559-bb6pp" podStartSLOduration=1.753201034 podStartE2EDuration="2.041237483s" totalImagesPullingTime="288.036449ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:07:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:07:27.019430056 +0000 UTC m=+1376.243249982" watchObservedRunningTime="2026-10-30 05:07:27.041237483 +0000 UTC m=+1376.265057385"
Oct 30 05:07:27.286723 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:27.286688    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8876-e8f1d5"
Oct 30 05:07:27.410450 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:27.410409    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cfa3691-14ae-46c8-9b12-5a6c34dd1562-kube-api-access-wf8b6\" (UniqueName: \"kubernetes.io/projected/5cfa3691-14ae-46c8-9b12-5a6c34dd1562-kube-api-access-wf8b6\") pod \"5cfa3691-14ae-46c8-9b12-5a6c34dd1562\" (UID: \"5cfa3691-14ae-46c8-9b12-5a6c34dd1562\") "
Oct 30 05:07:27.412786 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:27.412711    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cfa3691-14ae-46c8-9b12-5a6c34dd1562-kube-api-access-wf8b6" pod "5cfa3691-14ae-46c8-9b12-5a6c34dd1562" (UID: "5cfa3691-14ae-46c8-9b12-5a6c34dd1562"). InnerVolumeSpecName "kube-api-access-wf8b6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:07:27.513092 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:27.513058    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wf8b6\" (UniqueName: \"kubernetes.io/projected/5cfa3691-14ae-46c8-9b12-5a6c34dd1562-kube-api-access-wf8b6\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:07:27.714219 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:27.714150    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:07:27.988691 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:27.988596    5898 generic.go:356] "Generic (PLEG): container finished" podID="5cfa3691-14ae-46c8-9b12-5a6c34dd1562" containerID="8177ce25451a61f708b186374ab576934c14f9ebf1819c946ff94fa120afb02f" exitCode=137
Oct 30 05:07:27.988691 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:27.988663    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8876-e8f1d5"
Oct 30 05:07:27.988929 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:27.988695    5898 scope.go:122] "RemoveContainer" containerID="8177ce25451a61f708b186374ab576934c14f9ebf1819c946ff94fa120afb02f"
Oct 30 05:07:28.000359 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:28.000341    5898 scope.go:122] "RemoveContainer" containerID="8177ce25451a61f708b186374ab576934c14f9ebf1819c946ff94fa120afb02f"
Oct 30 05:07:28.000642 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:07:28.000621    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8177ce25451a61f708b186374ab576934c14f9ebf1819c946ff94fa120afb02f\": container with ID starting with 8177ce25451a61f708b186374ab576934c14f9ebf1819c946ff94fa120afb02f not found: ID does not exist" containerID="8177ce25451a61f708b186374ab576934c14f9ebf1819c946ff94fa120afb02f"
Oct 30 05:07:28.000690 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:28.000651    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8177ce25451a61f708b186374ab576934c14f9ebf1819c946ff94fa120afb02f"} err="failed to get container status \"8177ce25451a61f708b186374ab576934c14f9ebf1819c946ff94fa120afb02f\": rpc error: code = NotFound desc = could not find container \"8177ce25451a61f708b186374ab576934c14f9ebf1819c946ff94fa120afb02f\": container with ID starting with 8177ce25451a61f708b186374ab576934c14f9ebf1819c946ff94fa120afb02f not found: ID does not exist"
Oct 30 05:07:28.011693 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:07:28.011666    5898 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8876-e8f1d5\" is forbidden: User \"system:node:ip-10-0-135-98.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-98.ec2.internal' and this object" podUID="5cfa3691-14ae-46c8-9b12-5a6c34dd1562" pod="opendatahub/test-curl-8876-e8f1d5"
Oct 30 05:07:28.892417 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:28.892381    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cfa3691-14ae-46c8-9b12-5a6c34dd1562" path="/var/lib/kubelet/pods/5cfa3691-14ae-46c8-9b12-5a6c34dd1562/volumes"
Oct 30 05:07:29.461872 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:29.461840    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:07:31.252177 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:31.252142    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dfb231-8574fd944b-hd2hq"]
Oct 30 05:07:31.258034 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:31.258012    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dfb231-8574fd944b-hd2hq"
Oct 30 05:07:31.278391 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:31.278362    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dfb231-8574fd944b-hd2hq"]
Oct 30 05:07:31.287262 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:31.284611    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-dfb231-serving-cert\""
Oct 30 05:07:31.337881 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:31.337848    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:07:31.377853 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:31.377810    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b35f68e0-c492-44ba-a145-026a63713b60-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-dfb231-8574fd944b-hd2hq\" (UID: \"b35f68e0-c492-44ba-a145-026a63713b60\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dfb231-8574fd944b-hd2hq"
Oct 30 05:07:31.378050 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:31.377954    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpqh5\" (UniqueName: \"kubernetes.io/projected/b35f68e0-c492-44ba-a145-026a63713b60-kube-api-access-rpqh5\") pod \"maas-api-e2e-shared-b-w0-dfb231-8574fd944b-hd2hq\" (UID: \"b35f68e0-c492-44ba-a145-026a63713b60\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dfb231-8574fd944b-hd2hq"
Oct 30 05:07:31.378176 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:31.378151    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b35f68e0-c492-44ba-a145-026a63713b60-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-dfb231-8574fd944b-hd2hq\" (UID: \"b35f68e0-c492-44ba-a145-026a63713b60\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dfb231-8574fd944b-hd2hq"
Oct 30 05:07:31.378289 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:31.378259    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b35f68e0-c492-44ba-a145-026a63713b60-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-dfb231-8574fd944b-hd2hq\" (UID: \"b35f68e0-c492-44ba-a145-026a63713b60\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dfb231-8574fd944b-hd2hq"
Oct 30 05:07:31.432230 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:31.432183    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:07:31.479606 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:31.479576    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b35f68e0-c492-44ba-a145-026a63713b60-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-dfb231-8574fd944b-hd2hq\" (UID: \"b35f68e0-c492-44ba-a145-026a63713b60\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dfb231-8574fd944b-hd2hq"
Oct 30 05:07:31.479792 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:31.479682    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpqh5\" (UniqueName: \"kubernetes.io/projected/b35f68e0-c492-44ba-a145-026a63713b60-kube-api-access-rpqh5\") pod \"maas-api-e2e-shared-b-w0-dfb231-8574fd944b-hd2hq\" (UID: \"b35f68e0-c492-44ba-a145-026a63713b60\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dfb231-8574fd944b-hd2hq"
Oct 30 05:07:31.479873 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:31.479850    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b35f68e0-c492-44ba-a145-026a63713b60-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-dfb231-8574fd944b-hd2hq\" (UID: \"b35f68e0-c492-44ba-a145-026a63713b60\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dfb231-8574fd944b-hd2hq"
Oct 30 05:07:31.480144 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:31.479995    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b35f68e0-c492-44ba-a145-026a63713b60-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-dfb231-8574fd944b-hd2hq\" (UID: \"b35f68e0-c492-44ba-a145-026a63713b60\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dfb231-8574fd944b-hd2hq"
Oct 30 05:07:31.483945 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:31.483850    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b35f68e0-c492-44ba-a145-026a63713b60-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-dfb231-8574fd944b-hd2hq\" (UID: \"b35f68e0-c492-44ba-a145-026a63713b60\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dfb231-8574fd944b-hd2hq"
Oct 30 05:07:31.489350 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:31.486228    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b35f68e0-c492-44ba-a145-026a63713b60-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-dfb231-8574fd944b-hd2hq\" (UID: \"b35f68e0-c492-44ba-a145-026a63713b60\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dfb231-8574fd944b-hd2hq"
Oct 30 05:07:31.497955 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:31.494103    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b35f68e0-c492-44ba-a145-026a63713b60-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-dfb231-8574fd944b-hd2hq\" (UID: \"b35f68e0-c492-44ba-a145-026a63713b60\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dfb231-8574fd944b-hd2hq"
Oct 30 05:07:31.518976 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:31.512401    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpqh5\" (UniqueName: \"kubernetes.io/projected/b35f68e0-c492-44ba-a145-026a63713b60-kube-api-access-rpqh5\") pod \"maas-api-e2e-shared-b-w0-dfb231-8574fd944b-hd2hq\" (UID: \"b35f68e0-c492-44ba-a145-026a63713b60\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dfb231-8574fd944b-hd2hq"
Oct 30 05:07:31.540229 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:31.540191    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:07:31.572270 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:31.572208    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dfb231-8574fd944b-hd2hq"
Oct 30 05:07:31.889398 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:07:31.889368    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb35f68e0_c492_44ba_a145_026a63713b60.slice/crio-fd7620073d2ddd17093c5c4d3a90fb5dd538e04bccb11074a3b9a27437665beb WatchSource:0}: Error finding container fd7620073d2ddd17093c5c4d3a90fb5dd538e04bccb11074a3b9a27437665beb: Status 404 returned error can't find the container with id fd7620073d2ddd17093c5c4d3a90fb5dd538e04bccb11074a3b9a27437665beb
Oct 30 05:07:31.890655 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:31.890622    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dfb231-8574fd944b-hd2hq"]
Oct 30 05:07:32.026172 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:32.026137    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dfb231-8574fd944b-hd2hq" event={"ID":"b35f68e0-c492-44ba-a145-026a63713b60","Type":"ContainerStarted","Data":"fd7620073d2ddd17093c5c4d3a90fb5dd538e04bccb11074a3b9a27437665beb"}
Oct 30 05:07:33.027563 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:33.027533    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f2f688d-sw2jc"]
Oct 30 05:07:33.036271 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:33.036240    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dfb231-8574fd944b-hd2hq" event={"ID":"b35f68e0-c492-44ba-a145-026a63713b60","Type":"ContainerStarted","Data":"b9f2c05280b349e089bbe5d1658649c0c78a48f7e380fc6e1bad98b6404ab663"}
Oct 30 05:07:33.036445 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:33.036371    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dfb231-8574fd944b-hd2hq"
Oct 30 05:07:33.039404 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:33.039380    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f2f688d-sw2jc"]
Oct 30 05:07:33.076964 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:33.076912    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dfb231-8574fd944b-hd2hq" podStartSLOduration=1.7559831479999999 podStartE2EDuration="2.07688404s" totalImagesPullingTime="320.900892ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:07:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:07:33.072855972 +0000 UTC m=+1382.296675872" watchObservedRunningTime="2026-10-30 05:07:33.07688404 +0000 UTC m=+1382.300703939"
Oct 30 05:07:33.253503 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:33.253467    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:07:34.003310 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:34.003279    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef844616-6b67dd9559-bb6pp"
Oct 30 05:07:34.892778 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:34.892720    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6366537-f230-4971-b69b-ac4ffd5eb4c5" path="/var/lib/kubelet/pods/f6366537-f230-4971-b69b-ac4ffd5eb4c5/volumes"
Oct 30 05:07:34.922847 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:34.922822    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-6mdvj"]
Oct 30 05:07:35.013415 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:35.013380    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:07:35.112942 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:35.112894    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:07:36.944912 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:36.944877    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ac96f3-7846ddb485-qfvcp"]
Oct 30 05:07:36.949546 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:36.949529    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac96f3-7846ddb485-qfvcp"
Oct 30 05:07:36.967344 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:36.967316    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ac96f3-7846ddb485-qfvcp"]
Oct 30 05:07:36.969315 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:36.969292    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8c6428a-4956-430a-a835-8206dd6dcf9a-maas-api-tls\") pod \"maas-api-e2e-inf-a-ac96f3-7846ddb485-qfvcp\" (UID: \"a8c6428a-4956-430a-a835-8206dd6dcf9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac96f3-7846ddb485-qfvcp"
Oct 30 05:07:36.969467 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:36.969450    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8c6428a-4956-430a-a835-8206dd6dcf9a-metrics-certs\") pod \"maas-api-e2e-inf-a-ac96f3-7846ddb485-qfvcp\" (UID: \"a8c6428a-4956-430a-a835-8206dd6dcf9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac96f3-7846ddb485-qfvcp"
Oct 30 05:07:36.969635 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:36.969620    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vxr8\" (UniqueName: \"kubernetes.io/projected/a8c6428a-4956-430a-a835-8206dd6dcf9a-kube-api-access-4vxr8\") pod \"maas-api-e2e-inf-a-ac96f3-7846ddb485-qfvcp\" (UID: \"a8c6428a-4956-430a-a835-8206dd6dcf9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac96f3-7846ddb485-qfvcp"
Oct 30 05:07:36.969737 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:36.969661    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8c6428a-4956-430a-a835-8206dd6dcf9a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ac96f3-7846ddb485-qfvcp\" (UID: \"a8c6428a-4956-430a-a835-8206dd6dcf9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac96f3-7846ddb485-qfvcp"
Oct 30 05:07:36.974473 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:36.974451    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-ac96f3-serving-cert\""
Oct 30 05:07:37.021731 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:37.021699    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:07:37.070812 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:37.070772    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8c6428a-4956-430a-a835-8206dd6dcf9a-maas-api-tls\") pod \"maas-api-e2e-inf-a-ac96f3-7846ddb485-qfvcp\" (UID: \"a8c6428a-4956-430a-a835-8206dd6dcf9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac96f3-7846ddb485-qfvcp"
Oct 30 05:07:37.071018 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:37.070853    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8c6428a-4956-430a-a835-8206dd6dcf9a-metrics-certs\") pod \"maas-api-e2e-inf-a-ac96f3-7846ddb485-qfvcp\" (UID: \"a8c6428a-4956-430a-a835-8206dd6dcf9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac96f3-7846ddb485-qfvcp"
Oct 30 05:07:37.071018 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:37.070987    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vxr8\" (UniqueName: \"kubernetes.io/projected/a8c6428a-4956-430a-a835-8206dd6dcf9a-kube-api-access-4vxr8\") pod \"maas-api-e2e-inf-a-ac96f3-7846ddb485-qfvcp\" (UID: \"a8c6428a-4956-430a-a835-8206dd6dcf9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac96f3-7846ddb485-qfvcp"
Oct 30 05:07:37.071018 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:37.071014    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8c6428a-4956-430a-a835-8206dd6dcf9a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ac96f3-7846ddb485-qfvcp\" (UID: \"a8c6428a-4956-430a-a835-8206dd6dcf9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac96f3-7846ddb485-qfvcp"
Oct 30 05:07:37.071824 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:37.071793    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8c6428a-4956-430a-a835-8206dd6dcf9a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ac96f3-7846ddb485-qfvcp\" (UID: \"a8c6428a-4956-430a-a835-8206dd6dcf9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac96f3-7846ddb485-qfvcp"
Oct 30 05:07:37.073554 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:37.073529    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8c6428a-4956-430a-a835-8206dd6dcf9a-maas-api-tls\") pod \"maas-api-e2e-inf-a-ac96f3-7846ddb485-qfvcp\" (UID: \"a8c6428a-4956-430a-a835-8206dd6dcf9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac96f3-7846ddb485-qfvcp"
Oct 30 05:07:37.073655 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:37.073591    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8c6428a-4956-430a-a835-8206dd6dcf9a-metrics-certs\") pod \"maas-api-e2e-inf-a-ac96f3-7846ddb485-qfvcp\" (UID: \"a8c6428a-4956-430a-a835-8206dd6dcf9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac96f3-7846ddb485-qfvcp"
Oct 30 05:07:37.089538 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:37.089516    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vxr8\" (UniqueName: \"kubernetes.io/projected/a8c6428a-4956-430a-a835-8206dd6dcf9a-kube-api-access-4vxr8\") pod \"maas-api-e2e-inf-a-ac96f3-7846ddb485-qfvcp\" (UID: \"a8c6428a-4956-430a-a835-8206dd6dcf9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac96f3-7846ddb485-qfvcp"
Oct 30 05:07:37.112294 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:37.112271    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:07:37.212161 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:37.212080    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:07:37.267265 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:37.267227    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac96f3-7846ddb485-qfvcp"
Oct 30 05:07:37.589752 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:37.589719    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ac96f3-7846ddb485-qfvcp"]
Oct 30 05:07:37.589864 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:07:37.589765    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8c6428a_4956_430a_a835_8206dd6dcf9a.slice/crio-358752aac99eb1b9111761c964c33854cb8e7ec6433163db64fabfe2349e1b9c WatchSource:0}: Error finding container 358752aac99eb1b9111761c964c33854cb8e7ec6433163db64fabfe2349e1b9c: Status 404 returned error can't find the container with id 358752aac99eb1b9111761c964c33854cb8e7ec6433163db64fabfe2349e1b9c
Oct 30 05:07:38.083083 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:38.083052    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac96f3-7846ddb485-qfvcp" event={"ID":"a8c6428a-4956-430a-a835-8206dd6dcf9a","Type":"ContainerStarted","Data":"358752aac99eb1b9111761c964c33854cb8e7ec6433163db64fabfe2349e1b9c"}
Oct 30 05:07:39.053952 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:39.053923    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dfb231-8574fd944b-hd2hq"
Oct 30 05:07:39.093116 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:39.093076    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:07:39.093934 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:39.093828    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac96f3-7846ddb485-qfvcp" event={"ID":"a8c6428a-4956-430a-a835-8206dd6dcf9a","Type":"ContainerStarted","Data":"49458ce621e18223c3d30525ff077aa3b54d1d8ea8697748869d6ab71185fd72"}
Oct 30 05:07:39.094051 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:39.093995    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac96f3-7846ddb485-qfvcp"
Oct 30 05:07:39.138919 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:39.138846    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac96f3-7846ddb485-qfvcp" podStartSLOduration=2.712301796 podStartE2EDuration="3.138829159s" totalImagesPullingTime="426.527363ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:07:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:07:39.131687271 +0000 UTC m=+1388.355507170" watchObservedRunningTime="2026-10-30 05:07:39.138829159 +0000 UTC m=+1388.362649064"
Oct 30 05:07:39.818920 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:39.818867    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:07:39.924126 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:39.924090    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-6mdvj" podUID="816aafb6-0c1d-419f-9c95-191371977ef4" containerName="payload-pre-processing" containerID="cri-o://f99809889cb27825411ba243b1b86eb6caafbb2b34437c8f99b246086bd1a0c9" gracePeriod=25
Oct 30 05:07:40.834857 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:40.834824    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:07:41.038333 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:41.038301    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed8db1e0-t48w9"]
Oct 30 05:07:41.049946 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:41.049914    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed8db1e0-t48w9"]
Oct 30 05:07:42.892044 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:42.892012    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7804a436-264d-4812-8446-7ce83590a9b7" path="/var/lib/kubelet/pods/7804a436-264d-4812-8446-7ce83590a9b7/volumes"
Oct 30 05:07:43.029206 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:43.029172    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:07:44.030987 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:44.030953    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6447008-d4gbk"]
Oct 30 05:07:44.042053 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:44.042023    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6447008-d4gbk"]
Oct 30 05:07:44.891882 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:44.891850    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a171566-49dd-491e-8f7b-a07f07c15453" path="/var/lib/kubelet/pods/5a171566-49dd-491e-8f7b-a07f07c15453/volumes"
Oct 30 05:07:44.952466 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:44.952434    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:07:45.109512 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:45.109463    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac96f3-7846ddb485-qfvcp"
Oct 30 05:07:45.840492 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:45.840454    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:07:46.885214 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:46.885172    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:07:47.711543 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:47.711505    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-6mdvj" podUID="816aafb6-0c1d-419f-9c95-191371977ef4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:07:48.478742 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:48.478706    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:07:50.494722 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:50.494687    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:07:50.610923 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:50.608464    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:07:52.230801 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:52.230760    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:07:52.603572 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:52.603538    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:07:53.254284 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:53.254233    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:07:53.742125 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:53.742088    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:07:53.893521 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:53.893484    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:07:54.081874 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:54.081833    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bdaa2cf-5k489"]
Oct 30 05:07:54.100309 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:54.100260    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bdaa2cf-5k489"]
Oct 30 05:07:54.198921 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:54.198134    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:07:54.552006 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:54.551860    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:07:54.900434 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:54.900397    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:07:54.929298 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:54.929262    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55c4c3a3-a899-4431-8a8b-44382fec5062" path="/var/lib/kubelet/pods/55c4c3a3-a899-4431-8a8b-44382fec5062/volumes"
Oct 30 05:07:56.279924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:56.279583    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:07:56.854309 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:07:56.854273    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:07:57.711669 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:57.711616    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-6mdvj" podUID="816aafb6-0c1d-419f-9c95-191371977ef4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:07:58.512109 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:58.512047    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:07:59.116037 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:59.115973    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:07:59.228932 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:07:59.228883    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:00.239647 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:00.239620    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-6mdvj"
Oct 30 05:08:00.300889 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:00.300854    5898 generic.go:356] "Generic (PLEG): container finished" podID="816aafb6-0c1d-419f-9c95-191371977ef4" containerID="f99809889cb27825411ba243b1b86eb6caafbb2b34437c8f99b246086bd1a0c9" exitCode=0
Oct 30 05:08:00.301088 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:00.300934    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-6mdvj" event={"ID":"816aafb6-0c1d-419f-9c95-191371977ef4","Type":"ContainerDied","Data":"f99809889cb27825411ba243b1b86eb6caafbb2b34437c8f99b246086bd1a0c9"}
Oct 30 05:08:00.301088 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:00.300956    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-6mdvj"
Oct 30 05:08:00.301088 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:00.301001    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-6mdvj" event={"ID":"816aafb6-0c1d-419f-9c95-191371977ef4","Type":"ContainerDied","Data":"b5d6145c642e66b8a7275ee8d5fec726a444699492db898a520da78f751161a1"}
Oct 30 05:08:00.301088 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:00.301028    5898 scope.go:122] "RemoveContainer" containerID="f99809889cb27825411ba243b1b86eb6caafbb2b34437c8f99b246086bd1a0c9"
Oct 30 05:08:00.311784 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:00.311757    5898 scope.go:122] "RemoveContainer" containerID="f99809889cb27825411ba243b1b86eb6caafbb2b34437c8f99b246086bd1a0c9"
Oct 30 05:08:00.312363 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:08:00.312300    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f99809889cb27825411ba243b1b86eb6caafbb2b34437c8f99b246086bd1a0c9\": container with ID starting with f99809889cb27825411ba243b1b86eb6caafbb2b34437c8f99b246086bd1a0c9 not found: ID does not exist" containerID="f99809889cb27825411ba243b1b86eb6caafbb2b34437c8f99b246086bd1a0c9"
Oct 30 05:08:00.312363 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:00.312336    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f99809889cb27825411ba243b1b86eb6caafbb2b34437c8f99b246086bd1a0c9"} err="failed to get container status \"f99809889cb27825411ba243b1b86eb6caafbb2b34437c8f99b246086bd1a0c9\": rpc error: code = NotFound desc = could not find container \"f99809889cb27825411ba243b1b86eb6caafbb2b34437c8f99b246086bd1a0c9\": container with ID starting with f99809889cb27825411ba243b1b86eb6caafbb2b34437c8f99b246086bd1a0c9 not found: ID does not exist"
Oct 30 05:08:00.394518 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:00.394480    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/816aafb6-0c1d-419f-9c95-191371977ef4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/816aafb6-0c1d-419f-9c95-191371977ef4-plugins-config-volume\") pod \"816aafb6-0c1d-419f-9c95-191371977ef4\" (UID: \"816aafb6-0c1d-419f-9c95-191371977ef4\") "
Oct 30 05:08:00.394716 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:00.394532    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/816aafb6-0c1d-419f-9c95-191371977ef4-kube-api-access-hrzz2\" (UniqueName: \"kubernetes.io/projected/816aafb6-0c1d-419f-9c95-191371977ef4-kube-api-access-hrzz2\") pod \"816aafb6-0c1d-419f-9c95-191371977ef4\" (UID: \"816aafb6-0c1d-419f-9c95-191371977ef4\") "
Oct 30 05:08:00.394956 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:00.394929    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/816aafb6-0c1d-419f-9c95-191371977ef4-plugins-config-volume" pod "816aafb6-0c1d-419f-9c95-191371977ef4" (UID: "816aafb6-0c1d-419f-9c95-191371977ef4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:08:00.395200 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:00.395181    5898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/816aafb6-0c1d-419f-9c95-191371977ef4-plugins-config-volume\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:08:00.396693 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:00.396672    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/816aafb6-0c1d-419f-9c95-191371977ef4-kube-api-access-hrzz2" pod "816aafb6-0c1d-419f-9c95-191371977ef4" (UID: "816aafb6-0c1d-419f-9c95-191371977ef4"). InnerVolumeSpecName "kube-api-access-hrzz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:08:00.496578 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:00.496503    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrzz2\" (UniqueName: \"kubernetes.io/projected/816aafb6-0c1d-419f-9c95-191371977ef4-kube-api-access-hrzz2\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:08:00.650099 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:00.650063    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-6mdvj"]
Oct 30 05:08:00.661683 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:00.661647    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c613f5ea-55445d94bc-6mdvj"]
Oct 30 05:08:00.709809 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:00.709772    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:00.898641 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:00.898606    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="816aafb6-0c1d-419f-9c95-191371977ef4" path="/var/lib/kubelet/pods/816aafb6-0c1d-419f-9c95-191371977ef4/volumes"
Oct 30 05:08:00.975254 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:00.975208    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:01.089286 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:01.089253    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:01.207554 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:01.207474    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:01.350036 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:01.349998    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:02.320144 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:02.320044    5898 generic.go:356] "Generic (PLEG): container finished" podID="9eed8971-bba2-4f8d-9e3c-4f95f426dab9" containerID="dbf57086b26e517fa9da3dd64a6356e1d905c90726702ec0287a77262d56c01b" exitCode=0
Oct 30 05:08:02.320144 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:02.320113    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc" event={"ID":"9eed8971-bba2-4f8d-9e3c-4f95f426dab9","Type":"ContainerDied","Data":"dbf57086b26e517fa9da3dd64a6356e1d905c90726702ec0287a77262d56c01b"}
Oct 30 05:08:02.320531 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:02.320516    5898 scope.go:122] "RemoveContainer" containerID="dbf57086b26e517fa9da3dd64a6356e1d905c90726702ec0287a77262d56c01b"
Oct 30 05:08:02.322147 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:02.322120    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:08:02.322244 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:02.322189    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:08:02.322244 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:02.322222    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:08:03.290784 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:03.290754    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:03.351340 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:03.351296    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc" event={"ID":"9eed8971-bba2-4f8d-9e3c-4f95f426dab9","Type":"ContainerStarted","Data":"e0a823349e22e838b6a543a824b2cb4495c645f83d2616d5f6927cc41048b3d9"}
Oct 30 05:08:03.435099 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:03.435016    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:03.455329 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:03.455302    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc"
Oct 30 05:08:03.455499 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:03.455466    5898 patch_prober.go:29] interesting pod/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 30 05:08:03.455542 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:03.455513    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc" podUID="9eed8971-bba2-4f8d-9e3c-4f95f426dab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 30 05:08:03.757664 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:03.757609    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:04.029375 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:04.029276    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50c66ea3-fkmqx"]
Oct 30 05:08:04.035553 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:04.035526    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50c66ea3-fkmqx"]
Oct 30 05:08:04.455269 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:04.455239    5898 patch_prober.go:29] interesting pod/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 30 05:08:04.455711 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:04.455287    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc" podUID="9eed8971-bba2-4f8d-9e3c-4f95f426dab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 30 05:08:04.907433 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:04.907399    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1d68ab4-b486-4ce1-b92e-e6caf2391dd9" path="/var/lib/kubelet/pods/f1d68ab4-b486-4ce1-b92e-e6caf2391dd9/volumes"
Oct 30 05:08:05.455057 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:05.455025    5898 patch_prober.go:29] interesting pod/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 30 05:08:05.455241 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:05.455079    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc" podUID="9eed8971-bba2-4f8d-9e3c-4f95f426dab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 30 05:08:05.894943 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:05.890378    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:06.455004 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:06.454970    5898 patch_prober.go:29] interesting pod/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 30 05:08:06.455199 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:06.455028    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc" podUID="9eed8971-bba2-4f8d-9e3c-4f95f426dab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 30 05:08:07.455617 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:07.455583    5898 patch_prober.go:29] interesting pod/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 30 05:08:07.456065 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:07.455633    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc" podUID="9eed8971-bba2-4f8d-9e3c-4f95f426dab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 30 05:08:07.740986 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:07.740853    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:08.000279 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:08.000192    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:08.455072 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:08.455040    5898 patch_prober.go:29] interesting pod/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 30 05:08:08.455245 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:08.455090    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc" podUID="9eed8971-bba2-4f8d-9e3c-4f95f426dab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 30 05:08:08.643109 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:08.643074    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:08.744742 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:08.744649    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:09.278615 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:09.278581    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:09.460033 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:09.460001    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc"
Oct 30 05:08:09.460446 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:09.460432    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc"
Oct 30 05:08:09.461265 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:09.461246    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc"
Oct 30 05:08:10.012085 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:10.012036    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:11.378640 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:11.378600    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:12.486189 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:12.486152    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:12.629465 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:12.629428    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:13.218710 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:13.218676    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:13.479978 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:13.472075    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:14.577104 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:14.577058    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:14.793721 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:14.793689    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:14.993325 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:14.993240    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:15.098767 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:15.098738    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:15.960553 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:15.960525    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:16.105375 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:16.105343    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:16.318325 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:16.318252    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:16.740027 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:16.739990    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:16.849949 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:16.849863    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:16.975427 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:16.975395    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:17.135379 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:17.135351    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:19.169265 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:19.169226    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:19.580609 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:19.580569    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-288e01-86b8bc958-dhvhl"]
Oct 30 05:08:19.581349 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:19.581283    5898 cpu_manager.go:414] "containerMap: removing container" podUID="816aafb6-0c1d-419f-9c95-191371977ef4" containerName="payload-pre-processing"
Oct 30 05:08:19.581349 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:19.581300    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="816aafb6-0c1d-419f-9c95-191371977ef4" containerName="payload-pre-processing"
Oct 30 05:08:19.581562 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:19.581451    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="816aafb6-0c1d-419f-9c95-191371977ef4" containerName="payload-pre-processing"
Oct 30 05:08:19.586791 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:19.586770    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-288e01-86b8bc958-dhvhl"
Oct 30 05:08:19.605555 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:19.605531    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-288e01-86b8bc958-dhvhl"]
Oct 30 05:08:19.692258 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:19.692225    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-288e01\""
Oct 30 05:08:19.704994 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:19.704202    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-288e01-dockercfg-4f5xl\""
Oct 30 05:08:19.809828 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:19.809794    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c370781c-0dfa-4699-92ad-808c257a300d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-288e01-86b8bc958-dhvhl\" (UID: \"c370781c-0dfa-4699-92ad-808c257a300d\") " pod="openshift-ingress/payload-processing-e2e-api-a-288e01-86b8bc958-dhvhl"
Oct 30 05:08:19.809828 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:19.809834    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7825\" (UniqueName: \"kubernetes.io/projected/c370781c-0dfa-4699-92ad-808c257a300d-kube-api-access-k7825\") pod \"payload-processing-e2e-api-a-288e01-86b8bc958-dhvhl\" (UID: \"c370781c-0dfa-4699-92ad-808c257a300d\") " pod="openshift-ingress/payload-processing-e2e-api-a-288e01-86b8bc958-dhvhl"
Oct 30 05:08:19.911200 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:19.911112    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c370781c-0dfa-4699-92ad-808c257a300d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-288e01-86b8bc958-dhvhl\" (UID: \"c370781c-0dfa-4699-92ad-808c257a300d\") " pod="openshift-ingress/payload-processing-e2e-api-a-288e01-86b8bc958-dhvhl"
Oct 30 05:08:19.911376 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:19.911223    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7825\" (UniqueName: \"kubernetes.io/projected/c370781c-0dfa-4699-92ad-808c257a300d-kube-api-access-k7825\") pod \"payload-processing-e2e-api-a-288e01-86b8bc958-dhvhl\" (UID: \"c370781c-0dfa-4699-92ad-808c257a300d\") " pod="openshift-ingress/payload-processing-e2e-api-a-288e01-86b8bc958-dhvhl"
Oct 30 05:08:19.911888 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:19.911864    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c370781c-0dfa-4699-92ad-808c257a300d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-288e01-86b8bc958-dhvhl\" (UID: \"c370781c-0dfa-4699-92ad-808c257a300d\") " pod="openshift-ingress/payload-processing-e2e-api-a-288e01-86b8bc958-dhvhl"
Oct 30 05:08:19.930767 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:19.930734    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7825\" (UniqueName: \"kubernetes.io/projected/c370781c-0dfa-4699-92ad-808c257a300d-kube-api-access-k7825\") pod \"payload-processing-e2e-api-a-288e01-86b8bc958-dhvhl\" (UID: \"c370781c-0dfa-4699-92ad-808c257a300d\") " pod="openshift-ingress/payload-processing-e2e-api-a-288e01-86b8bc958-dhvhl"
Oct 30 05:08:20.203617 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:20.203541    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-288e01-86b8bc958-dhvhl"
Oct 30 05:08:20.308115 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:20.308085    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:20.504847 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:08:20.504801    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc370781c_0dfa_4699_92ad_808c257a300d.slice/crio-0fc7e29ea88bc99c7887188cf187b5c076959abb409d9c3f30f92de0cfc6f57a WatchSource:0}: Error finding container 0fc7e29ea88bc99c7887188cf187b5c076959abb409d9c3f30f92de0cfc6f57a: Status 404 returned error can't find the container with id 0fc7e29ea88bc99c7887188cf187b5c076959abb409d9c3f30f92de0cfc6f57a
Oct 30 05:08:20.505094 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:20.505073    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-288e01-86b8bc958-dhvhl"]
Oct 30 05:08:20.546195 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:20.546164    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-288e01-86b8bc958-dhvhl" event={"ID":"c370781c-0dfa-4699-92ad-808c257a300d","Type":"ContainerStarted","Data":"0fc7e29ea88bc99c7887188cf187b5c076959abb409d9c3f30f92de0cfc6f57a"}
Oct 30 05:08:21.042974 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:21.041597    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:21.167821 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:21.167787    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:21.443549 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:21.443512    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:21.555595 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:21.555557    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-288e01-86b8bc958-dhvhl" event={"ID":"c370781c-0dfa-4699-92ad-808c257a300d","Type":"ContainerStarted","Data":"987b39049437006c4ba864569a3ea2f273d89248030d214b705af63678623e24"}
Oct 30 05:08:21.555785 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:21.555623    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-288e01-86b8bc958-dhvhl"
Oct 30 05:08:21.596066 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:21.596018    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-288e01-86b8bc958-dhvhl" podStartSLOduration=2.596003451 podStartE2EDuration="2.596003451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:08:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:08:21.593456465 +0000 UTC m=+1430.817276366" watchObservedRunningTime="2026-10-30 05:08:21.596003451 +0000 UTC m=+1430.819823351"
Oct 30 05:08:21.960316 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:21.959807    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254"]
Oct 30 05:08:21.973524 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:21.972676    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254"
Oct 30 05:08:21.989370 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:21.989340    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254"]
Oct 30 05:08:22.000823 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:22.000572    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-4225df-openshift-default-dockercfg-n8vvh\""
Oct 30 05:08:22.015437 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:22.015407    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:22.047240 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:22.047202    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e58b1f9b-6669-485e-93a6-6ef4555b1c8f-istiod-ca-cert\") pod \"e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254\" (UID: \"e58b1f9b-6669-485e-93a6-6ef4555b1c8f\") " pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254"
Oct 30 05:08:22.047441 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:22.047419    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e58b1f9b-6669-485e-93a6-6ef4555b1c8f-istio-podinfo\") pod \"e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254\" (UID: \"e58b1f9b-6669-485e-93a6-6ef4555b1c8f\") " pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254"
Oct 30 05:08:22.047530 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:22.047459    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e58b1f9b-6669-485e-93a6-6ef4555b1c8f-istio-token\") pod \"e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254\" (UID: \"e58b1f9b-6669-485e-93a6-6ef4555b1c8f\") " pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254"
Oct 30 05:08:22.047596 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:22.047527    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58b1f9b-6669-485e-93a6-6ef4555b1c8f-workload-socket\") pod \"e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254\" (UID: \"e58b1f9b-6669-485e-93a6-6ef4555b1c8f\") " pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254"
Oct 30 05:08:22.047596 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:22.047582    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e58b1f9b-6669-485e-93a6-6ef4555b1c8f-istio-data\") pod \"e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254\" (UID: \"e58b1f9b-6669-485e-93a6-6ef4555b1c8f\") " pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254"
Oct 30 05:08:22.047710 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:22.047619    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e58b1f9b-6669-485e-93a6-6ef4555b1c8f-workload-certs\") pod \"e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254\" (UID: \"e58b1f9b-6669-485e-93a6-6ef4555b1c8f\") " pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254"
Oct 30 05:08:22.047807 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:22.047782    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58b1f9b-6669-485e-93a6-6ef4555b1c8f-credential-socket\") pod \"e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254\" (UID: \"e58b1f9b-6669-485e-93a6-6ef4555b1c8f\") " pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254"
Oct 30 05:08:22.047876 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:22.047814    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm7dr\" (UniqueName: \"kubernetes.io/projected/e58b1f9b-6669-485e-93a6-6ef4555b1c8f-kube-api-access-xm7dr\") pod \"e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254\" (UID: \"e58b1f9b-6669-485e-93a6-6ef4555b1c8f\") " pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254"
Oct 30 05:08:22.047967 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:22.047889    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e58b1f9b-6669-485e-93a6-6ef4555b1c8f-istio-envoy\") pod \"e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254\" (UID: \"e58b1f9b-6669-485e-93a6-6ef4555b1c8f\") " pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254"
Oct 30 05:08:22.155706 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:22.155220    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58b1f9b-6669-485e-93a6-6ef4555b1c8f-credential-socket\") pod \"e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254\" (UID: \"e58b1f9b-6669-485e-93a6-6ef4555b1c8f\") " pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254"
Oct 30 05:08:22.155706 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:22.155271    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm7dr\" (UniqueName: \"kubernetes.io/projected/e58b1f9b-6669-485e-93a6-6ef4555b1c8f-kube-api-access-xm7dr\") pod \"e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254\" (UID: \"e58b1f9b-6669-485e-93a6-6ef4555b1c8f\") " pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254"
Oct 30 05:08:22.155706 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:22.155467    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e58b1f9b-6669-485e-93a6-6ef4555b1c8f-istio-envoy\") pod \"e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254\" (UID: \"e58b1f9b-6669-485e-93a6-6ef4555b1c8f\") " pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254"
Oct 30 05:08:22.155706 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:22.155572    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e58b1f9b-6669-485e-93a6-6ef4555b1c8f-istiod-ca-cert\") pod \"e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254\" (UID: \"e58b1f9b-6669-485e-93a6-6ef4555b1c8f\") " pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254"
Oct 30 05:08:22.156094 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:22.155723    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e58b1f9b-6669-485e-93a6-6ef4555b1c8f-istio-podinfo\") pod \"e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254\" (UID: \"e58b1f9b-6669-485e-93a6-6ef4555b1c8f\") " pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254"
Oct 30 05:08:22.156094 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:22.155774    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e58b1f9b-6669-485e-93a6-6ef4555b1c8f-istio-token\") pod \"e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254\" (UID: \"e58b1f9b-6669-485e-93a6-6ef4555b1c8f\") " pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254"
Oct 30 05:08:22.156094 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:22.155810    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58b1f9b-6669-485e-93a6-6ef4555b1c8f-workload-socket\") pod \"e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254\" (UID: \"e58b1f9b-6669-485e-93a6-6ef4555b1c8f\") " pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254"
Oct 30 05:08:22.156094 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:22.155875    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e58b1f9b-6669-485e-93a6-6ef4555b1c8f-istio-data\") pod \"e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254\" (UID: \"e58b1f9b-6669-485e-93a6-6ef4555b1c8f\") " pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254"
Oct 30 05:08:22.156094 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:22.155931    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e58b1f9b-6669-485e-93a6-6ef4555b1c8f-workload-certs\") pod \"e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254\" (UID: \"e58b1f9b-6669-485e-93a6-6ef4555b1c8f\") " pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254"
Oct 30 05:08:22.159920 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:22.156568    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e58b1f9b-6669-485e-93a6-6ef4555b1c8f-workload-certs\") pod \"e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254\" (UID: \"e58b1f9b-6669-485e-93a6-6ef4555b1c8f\") " pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254"
Oct 30 05:08:22.159920 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:22.156854    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58b1f9b-6669-485e-93a6-6ef4555b1c8f-credential-socket\") pod \"e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254\" (UID: \"e58b1f9b-6669-485e-93a6-6ef4555b1c8f\") " pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254"
Oct 30 05:08:22.178926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:22.169678    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58b1f9b-6669-485e-93a6-6ef4555b1c8f-workload-socket\") pod \"e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254\" (UID: \"e58b1f9b-6669-485e-93a6-6ef4555b1c8f\") " pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254"
Oct 30 05:08:22.178926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:22.170259    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e58b1f9b-6669-485e-93a6-6ef4555b1c8f-istio-data\") pod \"e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254\" (UID: \"e58b1f9b-6669-485e-93a6-6ef4555b1c8f\") " pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254"
Oct 30 05:08:22.178926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:22.172821    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e58b1f9b-6669-485e-93a6-6ef4555b1c8f-istiod-ca-cert\") pod \"e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254\" (UID: \"e58b1f9b-6669-485e-93a6-6ef4555b1c8f\") " pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254"
Oct 30 05:08:22.187922 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:22.180640    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e58b1f9b-6669-485e-93a6-6ef4555b1c8f-istio-podinfo\") pod \"e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254\" (UID: \"e58b1f9b-6669-485e-93a6-6ef4555b1c8f\") " pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254"
Oct 30 05:08:22.199954 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:22.195197    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e58b1f9b-6669-485e-93a6-6ef4555b1c8f-istio-token\") pod \"e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254\" (UID: \"e58b1f9b-6669-485e-93a6-6ef4555b1c8f\") " pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254"
Oct 30 05:08:22.199954 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:22.197242    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e58b1f9b-6669-485e-93a6-6ef4555b1c8f-istio-envoy\") pod \"e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254\" (UID: \"e58b1f9b-6669-485e-93a6-6ef4555b1c8f\") " pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254"
Oct 30 05:08:22.199954 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:22.197859    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm7dr\" (UniqueName: \"kubernetes.io/projected/e58b1f9b-6669-485e-93a6-6ef4555b1c8f-kube-api-access-xm7dr\") pod \"e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254\" (UID: \"e58b1f9b-6669-485e-93a6-6ef4555b1c8f\") " pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254"
Oct 30 05:08:22.234002 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:22.223983    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:22.306486 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:22.305681    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254"
Oct 30 05:08:22.752417 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:08:22.752381    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode58b1f9b_6669_485e_93a6_6ef4555b1c8f.slice/crio-e59ded7f73eae7b0221cffc9b86f5cdbd7e20d5058d5fb6413384e329d68d96e WatchSource:0}: Error finding container e59ded7f73eae7b0221cffc9b86f5cdbd7e20d5058d5fb6413384e329d68d96e: Status 404 returned error can't find the container with id e59ded7f73eae7b0221cffc9b86f5cdbd7e20d5058d5fb6413384e329d68d96e
Oct 30 05:08:22.752836 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:22.752740    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254"]
Oct 30 05:08:22.755787 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:22.755749    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:08:22.755923 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:22.755831    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:08:22.755923 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:22.755878    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:08:23.331963 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:23.331915    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:23.590420 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:23.590324    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254" event={"ID":"e58b1f9b-6669-485e-93a6-6ef4555b1c8f","Type":"ContainerStarted","Data":"9e63a8060c174e9087ce4beb5186c8b62f469a6c3c981c2223382020f6fd7f98"}
Oct 30 05:08:23.590420 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:23.590383    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254" event={"ID":"e58b1f9b-6669-485e-93a6-6ef4555b1c8f","Type":"ContainerStarted","Data":"e59ded7f73eae7b0221cffc9b86f5cdbd7e20d5058d5fb6413384e329d68d96e"}
Oct 30 05:08:23.644498 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:23.644404    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254" podStartSLOduration=2.644372789 podStartE2EDuration="2.644372789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:08:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:08:23.642448822 +0000 UTC m=+1432.866268724" watchObservedRunningTime="2026-10-30 05:08:23.644372789 +0000 UTC m=+1432.868192690"
Oct 30 05:08:24.307449 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:24.307415    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254"
Oct 30 05:08:24.312942 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:24.312887    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254"
Oct 30 05:08:24.598225 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:24.598198    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254"
Oct 30 05:08:24.599159 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:24.599142    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254"
Oct 30 05:08:25.265093 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:25.265059    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:26.197019 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:26.196981    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:26.298704 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:26.298641    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:26.819936 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:08:26.819886    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:08:27.226654 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:27.226621    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:27.933195 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:27.933165    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:28.035613 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:28.035579    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:28.364918 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:28.364777    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:28.379059 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:28.377851    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4225df-6bd7847ff6-bdtpc"]
Oct 30 05:08:28.385300 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:28.385266    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4225df-6bd7847ff6-bdtpc"
Oct 30 05:08:28.390595 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:28.390568    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4225df-6bd7847ff6-bdtpc"]
Oct 30 05:08:28.406819 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:28.405749    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-4225df-dockercfg-5jfhp\""
Oct 30 05:08:28.414593 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:28.414216    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-4225df\""
Oct 30 05:08:28.512344 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:28.512304    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63f37c81-e93d-4e27-ad10-9665c4951666-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4225df-6bd7847ff6-bdtpc\" (UID: \"63f37c81-e93d-4e27-ad10-9665c4951666\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4225df-6bd7847ff6-bdtpc"
Oct 30 05:08:28.512581 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:28.512553    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzbw2\" (UniqueName: \"kubernetes.io/projected/63f37c81-e93d-4e27-ad10-9665c4951666-kube-api-access-qzbw2\") pod \"payload-processing-e2e-inf-b-4225df-6bd7847ff6-bdtpc\" (UID: \"63f37c81-e93d-4e27-ad10-9665c4951666\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4225df-6bd7847ff6-bdtpc"
Oct 30 05:08:28.614500 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:28.614458    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63f37c81-e93d-4e27-ad10-9665c4951666-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4225df-6bd7847ff6-bdtpc\" (UID: \"63f37c81-e93d-4e27-ad10-9665c4951666\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4225df-6bd7847ff6-bdtpc"
Oct 30 05:08:28.614699 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:28.614615    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzbw2\" (UniqueName: \"kubernetes.io/projected/63f37c81-e93d-4e27-ad10-9665c4951666-kube-api-access-qzbw2\") pod \"payload-processing-e2e-inf-b-4225df-6bd7847ff6-bdtpc\" (UID: \"63f37c81-e93d-4e27-ad10-9665c4951666\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4225df-6bd7847ff6-bdtpc"
Oct 30 05:08:28.615207 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:28.615160    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63f37c81-e93d-4e27-ad10-9665c4951666-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4225df-6bd7847ff6-bdtpc\" (UID: \"63f37c81-e93d-4e27-ad10-9665c4951666\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4225df-6bd7847ff6-bdtpc"
Oct 30 05:08:28.634585 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:28.634551    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzbw2\" (UniqueName: \"kubernetes.io/projected/63f37c81-e93d-4e27-ad10-9665c4951666-kube-api-access-qzbw2\") pod \"payload-processing-e2e-inf-b-4225df-6bd7847ff6-bdtpc\" (UID: \"63f37c81-e93d-4e27-ad10-9665c4951666\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4225df-6bd7847ff6-bdtpc"
Oct 30 05:08:28.719782 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:28.719752    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4225df-6bd7847ff6-bdtpc"
Oct 30 05:08:29.025442 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:29.025415    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4225df-6bd7847ff6-bdtpc"]
Oct 30 05:08:29.035296 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:29.035276    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36b03010-ss8rh"]
Oct 30 05:08:29.041549 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:29.041517    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36b03010-ss8rh"]
Oct 30 05:08:29.667687 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:29.667625    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4225df-6bd7847ff6-bdtpc" event={"ID":"63f37c81-e93d-4e27-ad10-9665c4951666","Type":"ContainerStarted","Data":"0a45b2f93b89eb67ac1bc4eb74b1ca4b2ac879501cdd832e4ce3aeb814130e54"}
Oct 30 05:08:29.670892 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:29.670863    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4225df-6bd7847ff6-bdtpc" event={"ID":"63f37c81-e93d-4e27-ad10-9665c4951666","Type":"ContainerStarted","Data":"2fa0d0aa59d4ebaee6fbe92d77d0af3bb3b6e8cc431093c2f03cd94fce1ab2ac"}
Oct 30 05:08:29.671037 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:29.670944    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-4225df-6bd7847ff6-bdtpc"
Oct 30 05:08:29.696876 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:29.696852    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:29.706805 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:29.706762    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-4225df-6bd7847ff6-bdtpc" podStartSLOduration=1.706750292 podStartE2EDuration="1.706750292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:08:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:08:29.70399384 +0000 UTC m=+1438.927813743" watchObservedRunningTime="2026-10-30 05:08:29.706750292 +0000 UTC m=+1438.930570193"
Oct 30 05:08:30.892628 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:30.892596    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="504ddefe-3772-418a-8e89-39c1090cf6db" path="/var/lib/kubelet/pods/504ddefe-3772-418a-8e89-39c1090cf6db/volumes"
Oct 30 05:08:31.316678 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:31.316586    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:31.763969 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:31.763846    5898 scope.go:122] "RemoveContainer" containerID="4a692d258bf70e73862e8a79b3ecdae74a516fb5f39128bee72338b91c4e1c60"
Oct 30 05:08:31.794863 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:31.794836    5898 scope.go:122] "RemoveContainer" containerID="7fbcc6327c15f43e1f3e1652793951571925438f19b9c1ce865e35c9349d6917"
Oct 30 05:08:31.844086 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:31.833956    5898 scope.go:122] "RemoveContainer" containerID="484e62f784148bd6b6ce30ac4f8d4d3536e0e4555542c2fb2b368e9a53f36f1a"
Oct 30 05:08:32.153823 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:32.133835    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:32.334468 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:32.334422    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:32.419936 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:32.419849    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:32.584030 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:32.584002    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-288e01-86b8bc958-dhvhl"
Oct 30 05:08:34.273068 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:34.273038    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:35.083622 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:35.083592    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:35.535511 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:35.535384    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:08:36.330223 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:36.330190    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:36.424614 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:36.424579    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:36.524132 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:36.524097    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:38.761523 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:38.761440    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:39.544921 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:39.543941    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:39.638744 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:39.638700    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:40.027923 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:40.024481    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:40.129358 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:40.129322    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:40.792926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:40.791979    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9" podUID="1647f909-5580-4219-a2e1-30b4cd994379" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:08:40.938984 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:40.928041    5898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6" podUID="b09804e3-8135-4f6d-ae34-c45d6da9728e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:08:41.125925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:41.116471    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49" podUID="05eef4eb-f379-4dc7-a947-123bd85a4968" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:08:41.125925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:41.116923    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6" podUID="47ff272a-ac5c-4f59-bf7b-5c0692d09386" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:08:41.125925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:41.117224    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/distinct-w5-0b21ba-kserve-7674c59666-m2sbd" podUID="24c6b2d4-3377-4079-8273-3f48064c3a98" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:08:41.127684 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:41.127650    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb" podUID="6f16bd6a-a7f0-49bb-844f-57b777bde2ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:08:41.136931 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:41.128755    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6" podUID="d30a7a86-f1da-451c-81dd-ebdbaeb4f8af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/ready\": context deadline exceeded"
Oct 30 05:08:41.207428 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:41.207385    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5" podUID="ebf8f4a2-8ac6-48ec-bdba-3f6a99653264" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 05:08:41.305894 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:41.292979    5898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6" podUID="47ff272a-ac5c-4f59-bf7b-5c0692d09386" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": context deadline exceeded"
Oct 30 05:08:41.305894 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:41.293740    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:41.458356 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:41.457724    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6" podUID="b09804e3-8135-4f6d-ae34-c45d6da9728e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:08:41.560315 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:41.560280    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:41.781981 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:41.749344    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:41.958952 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:41.948946    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-4225df-6bd7847ff6-bdtpc"
Oct 30 05:08:42.156093 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:42.155096    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6" podUID="47ff272a-ac5c-4f59-bf7b-5c0692d09386" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:08:42.156093 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:42.155570    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6"
Oct 30 05:08:42.214967 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:42.214823    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6"
Oct 30 05:08:43.038118 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:43.038041    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:43.144729 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:43.144697    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:44.116213 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:44.116183    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:46.017364 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:46.017319    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:47.216613 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:47.216578    5898 generic.go:356] "Generic (PLEG): container finished" podID="0ec75360-8dfb-45e6-85cc-267e154966f8" containerID="dde1623074ec63842dffea36914a2b3fc9704c56c5df4478d23b263b1b8c50f8" exitCode=0
Oct 30 05:08:47.217113 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:47.216653    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5" event={"ID":"0ec75360-8dfb-45e6-85cc-267e154966f8","Type":"ContainerDied","Data":"dde1623074ec63842dffea36914a2b3fc9704c56c5df4478d23b263b1b8c50f8"}
Oct 30 05:08:47.217160 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:47.217147    5898 scope.go:122] "RemoveContainer" containerID="dde1623074ec63842dffea36914a2b3fc9704c56c5df4478d23b263b1b8c50f8"
Oct 30 05:08:47.219070 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:47.219038    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:08:47.219196 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:47.219122    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:08:47.219196 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:47.219172    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:08:47.730053 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:47.730017    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5"
Oct 30 05:08:47.926636 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:47.926601    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:48.229826 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:48.229779    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5" event={"ID":"0ec75360-8dfb-45e6-85cc-267e154966f8","Type":"ContainerStarted","Data":"ea7341b122eebb48a4b4b17e748b6c0a9b798e32d75efae9cee6cc6e91be2f64"}
Oct 30 05:08:48.699175 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:48.699139    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5"
Oct 30 05:08:48.700558 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:48.700530    5898 patch_prober.go:29] interesting pod/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 05:08:48.700696 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:48.700584    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5" podUID="0ec75360-8dfb-45e6-85cc-267e154966f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 05:08:49.671987 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:49.671944    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:49.699330 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:49.699294    5898 patch_prober.go:29] interesting pod/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 05:08:49.699531 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:49.699357    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5" podUID="0ec75360-8dfb-45e6-85cc-267e154966f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 05:08:50.699046 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:50.699007    5898 patch_prober.go:29] interesting pod/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 05:08:50.699491 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:50.699085    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5" podUID="0ec75360-8dfb-45e6-85cc-267e154966f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 05:08:51.568953 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:51.568878    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:51.702993 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:51.702960    5898 patch_prober.go:29] interesting pod/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 05:08:51.703573 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:51.703535    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5" podUID="0ec75360-8dfb-45e6-85cc-267e154966f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 05:08:52.699322 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:52.699290    5898 patch_prober.go:29] interesting pod/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 05:08:52.699463 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:52.699354    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5" podUID="0ec75360-8dfb-45e6-85cc-267e154966f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 05:08:53.299854 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:53.299814    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:53.698299 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:53.698249    5898 patch_prober.go:29] interesting pod/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 05:08:53.698471 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:53.698327    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5" podUID="0ec75360-8dfb-45e6-85cc-267e154966f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 05:08:54.703284 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:54.703252    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5"
Oct 30 05:08:54.703682 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:54.703525    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5"
Oct 30 05:08:54.704346 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:54.704325    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5"
Oct 30 05:08:54.996026 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:54.995893    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:56.823776 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:08:56.823735    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:08:56.833951 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:56.833887    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:56.933541 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:56.933510    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:57.041026 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:57.040917    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:08:58.676307 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:08:58.676268    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:09:00.458797 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:00.458763    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:09:02.350404 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:02.350372    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:09:04.088447 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:04.088414    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:09:05.730980 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:05.730944    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:09:05.833881 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:05.833844    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:09:05.943368 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:05.943162    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:09:07.866465 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:07.866427    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:09:09.659771 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:09.659718    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:09:11.302022 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:11.301991    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:09:13.134359 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:13.134326    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:09:13.219735 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:13.219701    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:09:15.578883 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:15.578852    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:09:17.699732 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:17.699692    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:09:19.671087 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:19.671042    5898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 05:09:19.671594 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:19.671131    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5" podUID="b79b4fcb-1af2-424b-9b56-f6f9d40f88b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:09:20.067141 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:20.067052    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:09:22.036146 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:22.036106    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:09:23.859255 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:23.859222    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft"]
Oct 30 05:09:23.859709 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:23.859449    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft" podUID="7e29899b-8f40-4c21-a63b-2ce27d602dc3" containerName="istio-proxy" containerID="cri-o://6555b1b0099159bd01fd8616bb1280319a0e9e03a6133b9614046d590e1f477b" gracePeriod=30
Oct 30 05:09:24.018996 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:24.018924    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:09:25.941698 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:25.941654    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:09:26.102473 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:26.102433    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:09:26.781349 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:26.781319    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:09:26.820559 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:09:26.820526    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:09:27.249016 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:27.248979    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:09:27.353508 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:27.353477    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:09:27.463930 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:27.462046    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:09:28.279770 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:28.279728    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:09:29.177650 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:29.177616    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft"
Oct 30 05:09:29.202554 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:29.202518    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e29899b-8f40-4c21-a63b-2ce27d602dc3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e29899b-8f40-4c21-a63b-2ce27d602dc3-istio-podinfo\") pod \"7e29899b-8f40-4c21-a63b-2ce27d602dc3\" (UID: \"7e29899b-8f40-4c21-a63b-2ce27d602dc3\") "
Oct 30 05:09:29.202554 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:29.202558    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e29899b-8f40-4c21-a63b-2ce27d602dc3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e29899b-8f40-4c21-a63b-2ce27d602dc3-workload-socket\") pod \"7e29899b-8f40-4c21-a63b-2ce27d602dc3\" (UID: \"7e29899b-8f40-4c21-a63b-2ce27d602dc3\") "
Oct 30 05:09:29.202778 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:29.202582    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e29899b-8f40-4c21-a63b-2ce27d602dc3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e29899b-8f40-4c21-a63b-2ce27d602dc3-credential-socket\") pod \"7e29899b-8f40-4c21-a63b-2ce27d602dc3\" (UID: \"7e29899b-8f40-4c21-a63b-2ce27d602dc3\") "
Oct 30 05:09:29.202778 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:29.202718    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e29899b-8f40-4c21-a63b-2ce27d602dc3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e29899b-8f40-4c21-a63b-2ce27d602dc3-workload-certs\") pod \"7e29899b-8f40-4c21-a63b-2ce27d602dc3\" (UID: \"7e29899b-8f40-4c21-a63b-2ce27d602dc3\") "
Oct 30 05:09:29.202778 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:29.202764    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e29899b-8f40-4c21-a63b-2ce27d602dc3-kube-api-access-xxggk\" (UniqueName: \"kubernetes.io/projected/7e29899b-8f40-4c21-a63b-2ce27d602dc3-kube-api-access-xxggk\") pod \"7e29899b-8f40-4c21-a63b-2ce27d602dc3\" (UID: \"7e29899b-8f40-4c21-a63b-2ce27d602dc3\") "
Oct 30 05:09:29.202997 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:29.202801    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e29899b-8f40-4c21-a63b-2ce27d602dc3-workload-socket" pod "7e29899b-8f40-4c21-a63b-2ce27d602dc3" (UID: "7e29899b-8f40-4c21-a63b-2ce27d602dc3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:09:29.202997 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:29.202823    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e29899b-8f40-4c21-a63b-2ce27d602dc3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e29899b-8f40-4c21-a63b-2ce27d602dc3-istiod-ca-cert\") pod \"7e29899b-8f40-4c21-a63b-2ce27d602dc3\" (UID: \"7e29899b-8f40-4c21-a63b-2ce27d602dc3\") "
Oct 30 05:09:29.202997 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:29.202920    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e29899b-8f40-4c21-a63b-2ce27d602dc3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e29899b-8f40-4c21-a63b-2ce27d602dc3-istio-envoy\") pod \"7e29899b-8f40-4c21-a63b-2ce27d602dc3\" (UID: \"7e29899b-8f40-4c21-a63b-2ce27d602dc3\") "
Oct 30 05:09:29.203160 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:29.202990    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e29899b-8f40-4c21-a63b-2ce27d602dc3-credential-socket" pod "7e29899b-8f40-4c21-a63b-2ce27d602dc3" (UID: "7e29899b-8f40-4c21-a63b-2ce27d602dc3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:09:29.203160 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:29.203005    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e29899b-8f40-4c21-a63b-2ce27d602dc3-workload-certs" pod "7e29899b-8f40-4c21-a63b-2ce27d602dc3" (UID: "7e29899b-8f40-4c21-a63b-2ce27d602dc3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:09:29.203160 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:29.203062    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e29899b-8f40-4c21-a63b-2ce27d602dc3-istio-token\" (UniqueName: \"kubernetes.io/projected/7e29899b-8f40-4c21-a63b-2ce27d602dc3-istio-token\") pod \"7e29899b-8f40-4c21-a63b-2ce27d602dc3\" (UID: \"7e29899b-8f40-4c21-a63b-2ce27d602dc3\") "
Oct 30 05:09:29.203160 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:29.203102    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e29899b-8f40-4c21-a63b-2ce27d602dc3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e29899b-8f40-4c21-a63b-2ce27d602dc3-istio-data\") pod \"7e29899b-8f40-4c21-a63b-2ce27d602dc3\" (UID: \"7e29899b-8f40-4c21-a63b-2ce27d602dc3\") "
Oct 30 05:09:29.203385 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:29.203302    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e29899b-8f40-4c21-a63b-2ce27d602dc3-istiod-ca-cert" pod "7e29899b-8f40-4c21-a63b-2ce27d602dc3" (UID: "7e29899b-8f40-4c21-a63b-2ce27d602dc3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:09:29.204650 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:29.203700    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e29899b-8f40-4c21-a63b-2ce27d602dc3-istio-data" pod "7e29899b-8f40-4c21-a63b-2ce27d602dc3" (UID: "7e29899b-8f40-4c21-a63b-2ce27d602dc3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:09:29.204650 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:29.204004    5898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e29899b-8f40-4c21-a63b-2ce27d602dc3-istio-data\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:09:29.204650 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:29.204030    5898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e29899b-8f40-4c21-a63b-2ce27d602dc3-workload-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:09:29.204650 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:29.204046    5898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e29899b-8f40-4c21-a63b-2ce27d602dc3-credential-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:09:29.204650 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:29.204061    5898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e29899b-8f40-4c21-a63b-2ce27d602dc3-workload-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:09:29.204650 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:29.204075    5898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e29899b-8f40-4c21-a63b-2ce27d602dc3-istiod-ca-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:09:29.205204 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:29.205182    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e29899b-8f40-4c21-a63b-2ce27d602dc3-istio-envoy" pod "7e29899b-8f40-4c21-a63b-2ce27d602dc3" (UID: "7e29899b-8f40-4c21-a63b-2ce27d602dc3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:09:29.205369 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:29.205346    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e29899b-8f40-4c21-a63b-2ce27d602dc3-istio-podinfo" pod "7e29899b-8f40-4c21-a63b-2ce27d602dc3" (UID: "7e29899b-8f40-4c21-a63b-2ce27d602dc3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:09:29.205495 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:29.205479    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e29899b-8f40-4c21-a63b-2ce27d602dc3-istio-token" pod "7e29899b-8f40-4c21-a63b-2ce27d602dc3" (UID: "7e29899b-8f40-4c21-a63b-2ce27d602dc3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:09:29.205558 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:29.205504    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e29899b-8f40-4c21-a63b-2ce27d602dc3-kube-api-access-xxggk" pod "7e29899b-8f40-4c21-a63b-2ce27d602dc3" (UID: "7e29899b-8f40-4c21-a63b-2ce27d602dc3"). InnerVolumeSpecName "kube-api-access-xxggk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:09:29.305913 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:29.305871    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxggk\" (UniqueName: \"kubernetes.io/projected/7e29899b-8f40-4c21-a63b-2ce27d602dc3-kube-api-access-xxggk\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:09:29.305913 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:29.305908    5898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e29899b-8f40-4c21-a63b-2ce27d602dc3-istio-envoy\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:09:29.305913 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:29.305921    5898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e29899b-8f40-4c21-a63b-2ce27d602dc3-istio-token\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:09:29.306360 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:29.305953    5898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e29899b-8f40-4c21-a63b-2ce27d602dc3-istio-podinfo\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:09:29.663571 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:29.663534    5898 generic.go:356] "Generic (PLEG): container finished" podID="7e29899b-8f40-4c21-a63b-2ce27d602dc3" containerID="6555b1b0099159bd01fd8616bb1280319a0e9e03a6133b9614046d590e1f477b" exitCode=0
Oct 30 05:09:29.663571 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:29.663577    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft" event={"ID":"7e29899b-8f40-4c21-a63b-2ce27d602dc3","Type":"ContainerDied","Data":"6555b1b0099159bd01fd8616bb1280319a0e9e03a6133b9614046d590e1f477b"}
Oct 30 05:09:29.663809 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:29.663591    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft"
Oct 30 05:09:29.663809 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:29.663618    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft" event={"ID":"7e29899b-8f40-4c21-a63b-2ce27d602dc3","Type":"ContainerDied","Data":"4dd764ea64cbb6bf41c8a339dedf44048fce5f3b7691ab5ef277dd96767d307f"}
Oct 30 05:09:29.663809 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:29.663648    5898 scope.go:122] "RemoveContainer" containerID="6555b1b0099159bd01fd8616bb1280319a0e9e03a6133b9614046d590e1f477b"
Oct 30 05:09:29.676075 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:29.676053    5898 scope.go:122] "RemoveContainer" containerID="6555b1b0099159bd01fd8616bb1280319a0e9e03a6133b9614046d590e1f477b"
Oct 30 05:09:29.676326 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:09:29.676304    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6555b1b0099159bd01fd8616bb1280319a0e9e03a6133b9614046d590e1f477b\": container with ID starting with 6555b1b0099159bd01fd8616bb1280319a0e9e03a6133b9614046d590e1f477b not found: ID does not exist" containerID="6555b1b0099159bd01fd8616bb1280319a0e9e03a6133b9614046d590e1f477b"
Oct 30 05:09:29.676400 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:29.676340    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6555b1b0099159bd01fd8616bb1280319a0e9e03a6133b9614046d590e1f477b"} err="failed to get container status \"6555b1b0099159bd01fd8616bb1280319a0e9e03a6133b9614046d590e1f477b\": rpc error: code = NotFound desc = could not find container \"6555b1b0099159bd01fd8616bb1280319a0e9e03a6133b9614046d590e1f477b\": container with ID starting with 6555b1b0099159bd01fd8616bb1280319a0e9e03a6133b9614046d590e1f477b not found: ID does not exist"
Oct 30 05:09:29.727857 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:29.727825    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft"]
Oct 30 05:09:29.751464 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:29.751434    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-5l6ft"]
Oct 30 05:09:30.362305 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:30.362276    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:09:30.875003 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:30.874969    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:09:30.915274 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:30.915239    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e29899b-8f40-4c21-a63b-2ce27d602dc3" path="/var/lib/kubelet/pods/7e29899b-8f40-4c21-a63b-2ce27d602dc3/volumes"
Oct 30 05:09:30.978277 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:30.978234    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:09:31.942913 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:31.942880    5898 scope.go:122] "RemoveContainer" containerID="a5f08f41bd74d310ccf7ddbff896b60038a7e9953716f1efec072c890c0ec115"
Oct 30 05:09:31.953549 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:31.953529    5898 scope.go:122] "RemoveContainer" containerID="0f5f4b3269fb66dce2acc3a7f61bf0f6f9fe1e2a57558aac9042ed428c13659f"
Oct 30 05:09:31.963241 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:31.963221    5898 scope.go:122] "RemoveContainer" containerID="831f5ac3be0f661f540804bd487a42e7b3ef9093f1bc579dbf543ec7dd919c7f"
Oct 30 05:09:31.972932 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:31.972893    5898 scope.go:122] "RemoveContainer" containerID="9d210bd9f30d5060b51b5001c713bd60449aa63df62324c461b4f58759c515b2"
Oct 30 05:09:31.981785 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:31.981758    5898 scope.go:122] "RemoveContainer" containerID="b94100dba9aa5bbf8b6ee39d670d63c2f9b87d830b61d8b6192e99621bf335e1"
Oct 30 05:09:31.990751 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:31.990733    5898 scope.go:122] "RemoveContainer" containerID="33f9025b2af5244d87bec97ae4647509d69ff86c1f037723fa1ca8b12ed23826"
Oct 30 05:09:32.533520 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:32.533488    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:09:32.664379 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:32.664341    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:09:32.771105 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:32.771064    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:09:34.663071 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:34.663038    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:09:36.558191 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:36.558158    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:09:38.370844 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:38.370789    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:09:39.519296 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:39.519261    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:09:39.617862 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:39.617830    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:09:40.165535 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:40.163973    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:09:40.541686 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:40.541607    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:09:42.432780 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:42.432741    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:09:43.651605 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:43.651546    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:09:44.345452 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:44.345417    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:09:44.568552 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:44.568519    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:09:44.891776 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:44.891737    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:09:45.054532 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:45.054496    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:09:46.663874 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:46.663843    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:09:48.502188 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:48.502154    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:09:49.207304 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:49.207275    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:09:49.311218 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:49.311189    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:09:50.500919 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:50.495428    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:09:50.613220 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:50.604024    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:09:50.741385 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:50.741341    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:09:52.418395 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:52.418360    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:09:52.573456 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:52.573422    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:09:52.916647 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:52.915436    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:09:55.155863 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:55.155824    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:09:56.819886 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:09:56.819848    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:09:57.064361 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:57.064314    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:09:58.278659 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:58.278631    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:09:58.989594 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:09:58.989559    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:00.739193 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:00.739133    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:00.867444 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:00.867412    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:02.648345 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:02.648308    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:04.483887 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:04.483852    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:06.081928 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:06.081376    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:10:06.192177 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:06.192143    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:07.709004 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:07.708966    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ef844616-6b67dd9559-bb6pp"]
Oct 30 05:10:07.709465 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:07.709158    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef844616-6b67dd9559-bb6pp" podUID="5036975c-76af-4820-88b6-3c1b53400520" containerName="maas-api" containerID="cri-o://e953fab6ce022d52057c484db33d11419c52d25a53ceb6f53d7b2627ae53449b" gracePeriod=30
Oct 30 05:10:07.777552 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:07.777486    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:08.293443 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:08.293177    5898 generic.go:356] "Generic (PLEG): container finished" podID="5036975c-76af-4820-88b6-3c1b53400520" containerID="e953fab6ce022d52057c484db33d11419c52d25a53ceb6f53d7b2627ae53449b" exitCode=0
Oct 30 05:10:08.293443 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:08.293371    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef844616-6b67dd9559-bb6pp" event={"ID":"5036975c-76af-4820-88b6-3c1b53400520","Type":"ContainerDied","Data":"e953fab6ce022d52057c484db33d11419c52d25a53ceb6f53d7b2627ae53449b"}
Oct 30 05:10:08.344462 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:08.344428    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef844616-6b67dd9559-bb6pp"
Oct 30 05:10:08.495686 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:08.493702    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5036975c-76af-4820-88b6-3c1b53400520-kube-api-access-g2974\" (UniqueName: \"kubernetes.io/projected/5036975c-76af-4820-88b6-3c1b53400520-kube-api-access-g2974\") pod \"5036975c-76af-4820-88b6-3c1b53400520\" (UID: \"5036975c-76af-4820-88b6-3c1b53400520\") "
Oct 30 05:10:08.495686 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:08.493951    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5036975c-76af-4820-88b6-3c1b53400520-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5036975c-76af-4820-88b6-3c1b53400520-maas-api-tls\") pod \"5036975c-76af-4820-88b6-3c1b53400520\" (UID: \"5036975c-76af-4820-88b6-3c1b53400520\") "
Oct 30 05:10:08.495686 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:08.494028    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5036975c-76af-4820-88b6-3c1b53400520-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5036975c-76af-4820-88b6-3c1b53400520-openshift-service-ca\") pod \"5036975c-76af-4820-88b6-3c1b53400520\" (UID: \"5036975c-76af-4820-88b6-3c1b53400520\") "
Oct 30 05:10:08.495686 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:08.494167    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5036975c-76af-4820-88b6-3c1b53400520-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5036975c-76af-4820-88b6-3c1b53400520-metrics-certs\") pod \"5036975c-76af-4820-88b6-3c1b53400520\" (UID: \"5036975c-76af-4820-88b6-3c1b53400520\") "
Oct 30 05:10:08.504136 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:08.504099    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5036975c-76af-4820-88b6-3c1b53400520-openshift-service-ca" pod "5036975c-76af-4820-88b6-3c1b53400520" (UID: "5036975c-76af-4820-88b6-3c1b53400520"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:10:08.519922 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:08.512726    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5036975c-76af-4820-88b6-3c1b53400520-kube-api-access-g2974" pod "5036975c-76af-4820-88b6-3c1b53400520" (UID: "5036975c-76af-4820-88b6-3c1b53400520"). InnerVolumeSpecName "kube-api-access-g2974". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:10:08.562190 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:08.562134    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5036975c-76af-4820-88b6-3c1b53400520-maas-api-tls" pod "5036975c-76af-4820-88b6-3c1b53400520" (UID: "5036975c-76af-4820-88b6-3c1b53400520"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:10:08.564401 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:08.564357    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5036975c-76af-4820-88b6-3c1b53400520-metrics-certs" pod "5036975c-76af-4820-88b6-3c1b53400520" (UID: "5036975c-76af-4820-88b6-3c1b53400520"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:10:08.646927 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:08.600386    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2974\" (UniqueName: \"kubernetes.io/projected/5036975c-76af-4820-88b6-3c1b53400520-kube-api-access-g2974\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:10:08.646927 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:08.600423    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5036975c-76af-4820-88b6-3c1b53400520-maas-api-tls\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:10:08.646927 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:08.600439    5898 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5036975c-76af-4820-88b6-3c1b53400520-openshift-service-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:10:08.646927 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:08.600454    5898 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5036975c-76af-4820-88b6-3c1b53400520-metrics-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:10:08.876683 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:08.876647    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:09.250293 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:09.250260    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:09.308939 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:09.308872    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef844616-6b67dd9559-bb6pp" event={"ID":"5036975c-76af-4820-88b6-3c1b53400520","Type":"ContainerDied","Data":"ae9bdf0574eda35173ab1b51f06e8edc60bf5f823f14a9b00c1cd7f942cf790f"}
Oct 30 05:10:09.308939 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:09.308945    5898 scope.go:122] "RemoveContainer" containerID="e953fab6ce022d52057c484db33d11419c52d25a53ceb6f53d7b2627ae53449b"
Oct 30 05:10:09.309140 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:09.309066    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ef844616-6b67dd9559-bb6pp"
Oct 30 05:10:09.349494 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:09.349462    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:09.361052 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:09.361026    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ef844616-6b67dd9559-bb6pp"]
Oct 30 05:10:09.373407 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:09.373359    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ef844616-6b67dd9559-bb6pp"]
Oct 30 05:10:09.453670 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:09.453634    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:09.525229 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:09.525150    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd"]
Oct 30 05:10:09.525644 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:09.525614    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd" podUID="736153cf-ff1d-4f3e-b460-a62cd005c988" containerName="istio-proxy" containerID="cri-o://7b778252e921f65e90058efdae9bb9f5e44b78bdff216dda41722e2e5957c15d" gracePeriod=30
Oct 30 05:10:09.725584 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:09.725550    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:10.954964 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:10.954873    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5036975c-76af-4820-88b6-3c1b53400520" path="/var/lib/kubelet/pods/5036975c-76af-4820-88b6-3c1b53400520/volumes"
Oct 30 05:10:11.236744 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:11.231813    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:11.332283 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:11.332239    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:13.272180 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:13.272105    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:14.923644 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:14.923595    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd"
Oct 30 05:10:15.119457 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:15.119361    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/736153cf-ff1d-4f3e-b460-a62cd005c988-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/736153cf-ff1d-4f3e-b460-a62cd005c988-workload-certs\") pod \"736153cf-ff1d-4f3e-b460-a62cd005c988\" (UID: \"736153cf-ff1d-4f3e-b460-a62cd005c988\") "
Oct 30 05:10:15.119646 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:15.119456    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/736153cf-ff1d-4f3e-b460-a62cd005c988-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/736153cf-ff1d-4f3e-b460-a62cd005c988-workload-socket\") pod \"736153cf-ff1d-4f3e-b460-a62cd005c988\" (UID: \"736153cf-ff1d-4f3e-b460-a62cd005c988\") "
Oct 30 05:10:15.119646 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:15.119493    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/736153cf-ff1d-4f3e-b460-a62cd005c988-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/736153cf-ff1d-4f3e-b460-a62cd005c988-istio-podinfo\") pod \"736153cf-ff1d-4f3e-b460-a62cd005c988\" (UID: \"736153cf-ff1d-4f3e-b460-a62cd005c988\") "
Oct 30 05:10:15.119776 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:15.119650    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/736153cf-ff1d-4f3e-b460-a62cd005c988-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/736153cf-ff1d-4f3e-b460-a62cd005c988-istio-envoy\") pod \"736153cf-ff1d-4f3e-b460-a62cd005c988\" (UID: \"736153cf-ff1d-4f3e-b460-a62cd005c988\") "
Oct 30 05:10:15.119776 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:15.119724    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/736153cf-ff1d-4f3e-b460-a62cd005c988-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/736153cf-ff1d-4f3e-b460-a62cd005c988-istio-data\") pod \"736153cf-ff1d-4f3e-b460-a62cd005c988\" (UID: \"736153cf-ff1d-4f3e-b460-a62cd005c988\") "
Oct 30 05:10:15.119892 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:15.119800    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/736153cf-ff1d-4f3e-b460-a62cd005c988-istio-token\" (UniqueName: \"kubernetes.io/projected/736153cf-ff1d-4f3e-b460-a62cd005c988-istio-token\") pod \"736153cf-ff1d-4f3e-b460-a62cd005c988\" (UID: \"736153cf-ff1d-4f3e-b460-a62cd005c988\") "
Oct 30 05:10:15.119892 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:15.119845    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/736153cf-ff1d-4f3e-b460-a62cd005c988-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/736153cf-ff1d-4f3e-b460-a62cd005c988-istiod-ca-cert\") pod \"736153cf-ff1d-4f3e-b460-a62cd005c988\" (UID: \"736153cf-ff1d-4f3e-b460-a62cd005c988\") "
Oct 30 05:10:15.120014 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:15.119940    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/736153cf-ff1d-4f3e-b460-a62cd005c988-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/736153cf-ff1d-4f3e-b460-a62cd005c988-credential-socket\") pod \"736153cf-ff1d-4f3e-b460-a62cd005c988\" (UID: \"736153cf-ff1d-4f3e-b460-a62cd005c988\") "
Oct 30 05:10:15.120014 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:15.119982    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/736153cf-ff1d-4f3e-b460-a62cd005c988-kube-api-access-679r9\" (UniqueName: \"kubernetes.io/projected/736153cf-ff1d-4f3e-b460-a62cd005c988-kube-api-access-679r9\") pod \"736153cf-ff1d-4f3e-b460-a62cd005c988\" (UID: \"736153cf-ff1d-4f3e-b460-a62cd005c988\") "
Oct 30 05:10:15.131929 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:15.131883    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:15.133608 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:15.133580    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/736153cf-ff1d-4f3e-b460-a62cd005c988-credential-socket" pod "736153cf-ff1d-4f3e-b460-a62cd005c988" (UID: "736153cf-ff1d-4f3e-b460-a62cd005c988"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:10:15.134797 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:15.134551    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/736153cf-ff1d-4f3e-b460-a62cd005c988-workload-socket" pod "736153cf-ff1d-4f3e-b460-a62cd005c988" (UID: "736153cf-ff1d-4f3e-b460-a62cd005c988"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:10:15.154544 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:15.136593    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/736153cf-ff1d-4f3e-b460-a62cd005c988-istio-data" pod "736153cf-ff1d-4f3e-b460-a62cd005c988" (UID: "736153cf-ff1d-4f3e-b460-a62cd005c988"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:10:15.154544 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:15.139204    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/736153cf-ff1d-4f3e-b460-a62cd005c988-istiod-ca-cert" pod "736153cf-ff1d-4f3e-b460-a62cd005c988" (UID: "736153cf-ff1d-4f3e-b460-a62cd005c988"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:10:15.154544 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:15.139983    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/736153cf-ff1d-4f3e-b460-a62cd005c988-workload-certs" pod "736153cf-ff1d-4f3e-b460-a62cd005c988" (UID: "736153cf-ff1d-4f3e-b460-a62cd005c988"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:10:15.157019 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:15.156885    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/736153cf-ff1d-4f3e-b460-a62cd005c988-kube-api-access-679r9" pod "736153cf-ff1d-4f3e-b460-a62cd005c988" (UID: "736153cf-ff1d-4f3e-b460-a62cd005c988"). InnerVolumeSpecName "kube-api-access-679r9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:10:15.157409 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:15.157365    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/736153cf-ff1d-4f3e-b460-a62cd005c988-istio-podinfo" pod "736153cf-ff1d-4f3e-b460-a62cd005c988" (UID: "736153cf-ff1d-4f3e-b460-a62cd005c988"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:10:15.157675 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:15.157646    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/736153cf-ff1d-4f3e-b460-a62cd005c988-istio-envoy" pod "736153cf-ff1d-4f3e-b460-a62cd005c988" (UID: "736153cf-ff1d-4f3e-b460-a62cd005c988"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:10:15.158073 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:15.158050    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/736153cf-ff1d-4f3e-b460-a62cd005c988-istio-token" pod "736153cf-ff1d-4f3e-b460-a62cd005c988" (UID: "736153cf-ff1d-4f3e-b460-a62cd005c988"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:10:15.245806 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:15.245517    5898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/736153cf-ff1d-4f3e-b460-a62cd005c988-workload-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:10:15.245806 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:15.245552    5898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/736153cf-ff1d-4f3e-b460-a62cd005c988-istio-podinfo\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:10:15.245806 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:15.245568    5898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/736153cf-ff1d-4f3e-b460-a62cd005c988-istio-envoy\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:10:15.245806 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:15.245581    5898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/736153cf-ff1d-4f3e-b460-a62cd005c988-istio-data\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:10:15.245806 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:15.245594    5898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/736153cf-ff1d-4f3e-b460-a62cd005c988-istio-token\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:10:15.245806 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:15.245607    5898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/736153cf-ff1d-4f3e-b460-a62cd005c988-istiod-ca-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:10:15.245806 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:15.245620    5898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/736153cf-ff1d-4f3e-b460-a62cd005c988-credential-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:10:15.245806 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:15.245635    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-679r9\" (UniqueName: \"kubernetes.io/projected/736153cf-ff1d-4f3e-b460-a62cd005c988-kube-api-access-679r9\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:10:15.245806 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:15.245649    5898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/736153cf-ff1d-4f3e-b460-a62cd005c988-workload-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:10:15.435337 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:15.435245    5898 generic.go:356] "Generic (PLEG): container finished" podID="736153cf-ff1d-4f3e-b460-a62cd005c988" containerID="7b778252e921f65e90058efdae9bb9f5e44b78bdff216dda41722e2e5957c15d" exitCode=0
Oct 30 05:10:15.435337 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:15.435294    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd"
Oct 30 05:10:15.435337 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:15.435329    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd" event={"ID":"736153cf-ff1d-4f3e-b460-a62cd005c988","Type":"ContainerDied","Data":"7b778252e921f65e90058efdae9bb9f5e44b78bdff216dda41722e2e5957c15d"}
Oct 30 05:10:15.435626 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:15.435384    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd" event={"ID":"736153cf-ff1d-4f3e-b460-a62cd005c988","Type":"ContainerDied","Data":"1a90e73417e86428876242c0fb075c3d94b9598bd05771933a826d6d31816b4e"}
Oct 30 05:10:15.435626 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:15.435412    5898 scope.go:122] "RemoveContainer" containerID="7b778252e921f65e90058efdae9bb9f5e44b78bdff216dda41722e2e5957c15d"
Oct 30 05:10:15.454067 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:15.454045    5898 scope.go:122] "RemoveContainer" containerID="7b778252e921f65e90058efdae9bb9f5e44b78bdff216dda41722e2e5957c15d"
Oct 30 05:10:15.454395 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:10:15.454375    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b778252e921f65e90058efdae9bb9f5e44b78bdff216dda41722e2e5957c15d\": container with ID starting with 7b778252e921f65e90058efdae9bb9f5e44b78bdff216dda41722e2e5957c15d not found: ID does not exist" containerID="7b778252e921f65e90058efdae9bb9f5e44b78bdff216dda41722e2e5957c15d"
Oct 30 05:10:15.454479 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:15.454405    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b778252e921f65e90058efdae9bb9f5e44b78bdff216dda41722e2e5957c15d"} err="failed to get container status \"7b778252e921f65e90058efdae9bb9f5e44b78bdff216dda41722e2e5957c15d\": rpc error: code = NotFound desc = could not find container \"7b778252e921f65e90058efdae9bb9f5e44b78bdff216dda41722e2e5957c15d\": container with ID starting with 7b778252e921f65e90058efdae9bb9f5e44b78bdff216dda41722e2e5957c15d not found: ID does not exist"
Oct 30 05:10:15.500775 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:15.500746    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd"]
Oct 30 05:10:15.516044 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:15.516018    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-8ddnd"]
Oct 30 05:10:16.108791 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:16.108754    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:16.507689 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:16.507594    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:16.601117 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:16.601056    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:16.868573 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:16.868544    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc"]
Oct 30 05:10:16.869041 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:16.869007    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc" podUID="9eed8971-bba2-4f8d-9e3c-4f95f426dab9" containerName="istio-proxy" containerID="cri-o://e0a823349e22e838b6a543a824b2cb4495c645f83d2616d5f6927cc41048b3d9" gracePeriod=30
Oct 30 05:10:16.897605 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:16.897572    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:16.906436 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:16.906398    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="736153cf-ff1d-4f3e-b460-a62cd005c988" path="/var/lib/kubelet/pods/736153cf-ff1d-4f3e-b460-a62cd005c988/volumes"
Oct 30 05:10:17.139974 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:17.130847    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:17.702956 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:17.702923    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:18.046311 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:18.043853    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:18.140409 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:18.140380    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:18.230506 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:18.230472    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:18.332091 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:18.332050    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:19.149402 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:19.149344    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:20.487096 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:20.487059    5898 generic.go:356] "Generic (PLEG): container finished" podID="e58b1f9b-6669-485e-93a6-6ef4555b1c8f" containerID="9e63a8060c174e9087ce4beb5186c8b62f469a6c3c981c2223382020f6fd7f98" exitCode=0
Oct 30 05:10:20.487594 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:20.487124    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254" event={"ID":"e58b1f9b-6669-485e-93a6-6ef4555b1c8f","Type":"ContainerDied","Data":"9e63a8060c174e9087ce4beb5186c8b62f469a6c3c981c2223382020f6fd7f98"}
Oct 30 05:10:20.487594 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:20.487553    5898 scope.go:122] "RemoveContainer" containerID="9e63a8060c174e9087ce4beb5186c8b62f469a6c3c981c2223382020f6fd7f98"
Oct 30 05:10:20.489387 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:20.489360    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:10:20.489468 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:20.489429    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:10:20.489468 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:20.489464    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:10:21.204731 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:21.204694    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:21.497481 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:21.497386    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254" event={"ID":"e58b1f9b-6669-485e-93a6-6ef4555b1c8f","Type":"ContainerStarted","Data":"6df9de15797cfc9dcd85fb3b7070c035553b2c8084fed3b79174407ee0cc6fd4"}
Oct 30 05:10:21.993759 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:21.993726    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:22.111709 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:22.111675    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:22.242475 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:22.242446    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc"
Oct 30 05:10:22.308455 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:22.308424    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254"
Oct 30 05:10:22.310092 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:22.310056    5898 patch_prober.go:29] interesting pod/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 30 05:10:22.310225 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:22.310138    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254" podUID="e58b1f9b-6669-485e-93a6-6ef4555b1c8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 30 05:10:22.310684 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:22.310648    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eed8971-bba2-4f8d-9e3c-4f95f426dab9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eed8971-bba2-4f8d-9e3c-4f95f426dab9-workload-certs\") pod \"9eed8971-bba2-4f8d-9e3c-4f95f426dab9\" (UID: \"9eed8971-bba2-4f8d-9e3c-4f95f426dab9\") "
Oct 30 05:10:22.310805 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:22.310717    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eed8971-bba2-4f8d-9e3c-4f95f426dab9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eed8971-bba2-4f8d-9e3c-4f95f426dab9-credential-socket\") pod \"9eed8971-bba2-4f8d-9e3c-4f95f426dab9\" (UID: \"9eed8971-bba2-4f8d-9e3c-4f95f426dab9\") "
Oct 30 05:10:22.310805 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:22.310763    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9eed8971-bba2-4f8d-9e3c-4f95f426dab9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eed8971-bba2-4f8d-9e3c-4f95f426dab9-istiod-ca-cert\") pod \"9eed8971-bba2-4f8d-9e3c-4f95f426dab9\" (UID: \"9eed8971-bba2-4f8d-9e3c-4f95f426dab9\") "
Oct 30 05:10:22.310930 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:22.310804    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9eed8971-bba2-4f8d-9e3c-4f95f426dab9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eed8971-bba2-4f8d-9e3c-4f95f426dab9-istio-podinfo\") pod \"9eed8971-bba2-4f8d-9e3c-4f95f426dab9\" (UID: \"9eed8971-bba2-4f8d-9e3c-4f95f426dab9\") "
Oct 30 05:10:22.312718 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:22.311075    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9eed8971-bba2-4f8d-9e3c-4f95f426dab9-kube-api-access-jp5pc\" (UniqueName: \"kubernetes.io/projected/9eed8971-bba2-4f8d-9e3c-4f95f426dab9-kube-api-access-jp5pc\") pod \"9eed8971-bba2-4f8d-9e3c-4f95f426dab9\" (UID: \"9eed8971-bba2-4f8d-9e3c-4f95f426dab9\") "
Oct 30 05:10:22.312718 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:22.311126    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eed8971-bba2-4f8d-9e3c-4f95f426dab9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eed8971-bba2-4f8d-9e3c-4f95f426dab9-istio-data\") pod \"9eed8971-bba2-4f8d-9e3c-4f95f426dab9\" (UID: \"9eed8971-bba2-4f8d-9e3c-4f95f426dab9\") "
Oct 30 05:10:22.312718 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:22.311193    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9eed8971-bba2-4f8d-9e3c-4f95f426dab9-istio-token\" (UniqueName: \"kubernetes.io/projected/9eed8971-bba2-4f8d-9e3c-4f95f426dab9-istio-token\") pod \"9eed8971-bba2-4f8d-9e3c-4f95f426dab9\" (UID: \"9eed8971-bba2-4f8d-9e3c-4f95f426dab9\") "
Oct 30 05:10:22.312718 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:22.311320    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eed8971-bba2-4f8d-9e3c-4f95f426dab9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eed8971-bba2-4f8d-9e3c-4f95f426dab9-istio-envoy\") pod \"9eed8971-bba2-4f8d-9e3c-4f95f426dab9\" (UID: \"9eed8971-bba2-4f8d-9e3c-4f95f426dab9\") "
Oct 30 05:10:22.312718 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:22.311440    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eed8971-bba2-4f8d-9e3c-4f95f426dab9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eed8971-bba2-4f8d-9e3c-4f95f426dab9-workload-socket\") pod \"9eed8971-bba2-4f8d-9e3c-4f95f426dab9\" (UID: \"9eed8971-bba2-4f8d-9e3c-4f95f426dab9\") "
Oct 30 05:10:22.313157 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:22.313116    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eed8971-bba2-4f8d-9e3c-4f95f426dab9-workload-socket" pod "9eed8971-bba2-4f8d-9e3c-4f95f426dab9" (UID: "9eed8971-bba2-4f8d-9e3c-4f95f426dab9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:10:22.318531 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:22.318459    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eed8971-bba2-4f8d-9e3c-4f95f426dab9-istio-data" pod "9eed8971-bba2-4f8d-9e3c-4f95f426dab9" (UID: "9eed8971-bba2-4f8d-9e3c-4f95f426dab9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:10:22.318649 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:22.318598    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eed8971-bba2-4f8d-9e3c-4f95f426dab9-istio-token" pod "9eed8971-bba2-4f8d-9e3c-4f95f426dab9" (UID: "9eed8971-bba2-4f8d-9e3c-4f95f426dab9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:10:22.327570 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:22.327537    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eed8971-bba2-4f8d-9e3c-4f95f426dab9-credential-socket" pod "9eed8971-bba2-4f8d-9e3c-4f95f426dab9" (UID: "9eed8971-bba2-4f8d-9e3c-4f95f426dab9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:10:22.327782 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:22.327762    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eed8971-bba2-4f8d-9e3c-4f95f426dab9-workload-certs" pod "9eed8971-bba2-4f8d-9e3c-4f95f426dab9" (UID: "9eed8971-bba2-4f8d-9e3c-4f95f426dab9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:10:22.328732 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:22.328380    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9eed8971-bba2-4f8d-9e3c-4f95f426dab9-istiod-ca-cert" pod "9eed8971-bba2-4f8d-9e3c-4f95f426dab9" (UID: "9eed8971-bba2-4f8d-9e3c-4f95f426dab9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:10:22.328732 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:22.328632    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eed8971-bba2-4f8d-9e3c-4f95f426dab9-kube-api-access-jp5pc" pod "9eed8971-bba2-4f8d-9e3c-4f95f426dab9" (UID: "9eed8971-bba2-4f8d-9e3c-4f95f426dab9"). InnerVolumeSpecName "kube-api-access-jp5pc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:10:22.330421 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:22.330393    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eed8971-bba2-4f8d-9e3c-4f95f426dab9-istio-envoy" pod "9eed8971-bba2-4f8d-9e3c-4f95f426dab9" (UID: "9eed8971-bba2-4f8d-9e3c-4f95f426dab9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:10:22.361283 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:22.361252    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9eed8971-bba2-4f8d-9e3c-4f95f426dab9-istio-podinfo" pod "9eed8971-bba2-4f8d-9e3c-4f95f426dab9" (UID: "9eed8971-bba2-4f8d-9e3c-4f95f426dab9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:10:22.414752 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:22.414716    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jp5pc\" (UniqueName: \"kubernetes.io/projected/9eed8971-bba2-4f8d-9e3c-4f95f426dab9-kube-api-access-jp5pc\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:10:22.414752 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:22.414751    5898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eed8971-bba2-4f8d-9e3c-4f95f426dab9-istio-data\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:10:22.414994 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:22.414766    5898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eed8971-bba2-4f8d-9e3c-4f95f426dab9-istio-token\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:10:22.414994 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:22.414778    5898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eed8971-bba2-4f8d-9e3c-4f95f426dab9-istio-envoy\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:10:22.414994 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:22.414789    5898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eed8971-bba2-4f8d-9e3c-4f95f426dab9-workload-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:10:22.414994 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:22.414801    5898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eed8971-bba2-4f8d-9e3c-4f95f426dab9-workload-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:10:22.414994 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:22.414814    5898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eed8971-bba2-4f8d-9e3c-4f95f426dab9-credential-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:10:22.414994 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:22.414827    5898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eed8971-bba2-4f8d-9e3c-4f95f426dab9-istiod-ca-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:10:22.414994 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:22.414839    5898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eed8971-bba2-4f8d-9e3c-4f95f426dab9-istio-podinfo\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:10:22.508513 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:22.508420    5898 generic.go:356] "Generic (PLEG): container finished" podID="9eed8971-bba2-4f8d-9e3c-4f95f426dab9" containerID="e0a823349e22e838b6a543a824b2cb4495c645f83d2616d5f6927cc41048b3d9" exitCode=0
Oct 30 05:10:22.508513 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:22.508490    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc"
Oct 30 05:10:22.509055 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:22.508510    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc" event={"ID":"9eed8971-bba2-4f8d-9e3c-4f95f426dab9","Type":"ContainerDied","Data":"e0a823349e22e838b6a543a824b2cb4495c645f83d2616d5f6927cc41048b3d9"}
Oct 30 05:10:22.509055 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:22.508580    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc" event={"ID":"9eed8971-bba2-4f8d-9e3c-4f95f426dab9","Type":"ContainerDied","Data":"2933edd40042378c0b5ed6fa9c8afec2f1e807cdb6fa026851a597c41ee41d24"}
Oct 30 05:10:22.509055 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:22.508611    5898 scope.go:122] "RemoveContainer" containerID="e0a823349e22e838b6a543a824b2cb4495c645f83d2616d5f6927cc41048b3d9"
Oct 30 05:10:22.520313 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:22.520296    5898 scope.go:122] "RemoveContainer" containerID="dbf57086b26e517fa9da3dd64a6356e1d905c90726702ec0287a77262d56c01b"
Oct 30 05:10:22.530033 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:22.530015    5898 scope.go:122] "RemoveContainer" containerID="e0a823349e22e838b6a543a824b2cb4495c645f83d2616d5f6927cc41048b3d9"
Oct 30 05:10:22.530281 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:10:22.530261    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0a823349e22e838b6a543a824b2cb4495c645f83d2616d5f6927cc41048b3d9\": container with ID starting with e0a823349e22e838b6a543a824b2cb4495c645f83d2616d5f6927cc41048b3d9 not found: ID does not exist" containerID="e0a823349e22e838b6a543a824b2cb4495c645f83d2616d5f6927cc41048b3d9"
Oct 30 05:10:22.530362 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:22.530296    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0a823349e22e838b6a543a824b2cb4495c645f83d2616d5f6927cc41048b3d9"} err="failed to get container status \"e0a823349e22e838b6a543a824b2cb4495c645f83d2616d5f6927cc41048b3d9\": rpc error: code = NotFound desc = could not find container \"e0a823349e22e838b6a543a824b2cb4495c645f83d2616d5f6927cc41048b3d9\": container with ID starting with e0a823349e22e838b6a543a824b2cb4495c645f83d2616d5f6927cc41048b3d9 not found: ID does not exist"
Oct 30 05:10:22.530362 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:22.530318    5898 scope.go:122] "RemoveContainer" containerID="dbf57086b26e517fa9da3dd64a6356e1d905c90726702ec0287a77262d56c01b"
Oct 30 05:10:22.530560 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:10:22.530542    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbf57086b26e517fa9da3dd64a6356e1d905c90726702ec0287a77262d56c01b\": container with ID starting with dbf57086b26e517fa9da3dd64a6356e1d905c90726702ec0287a77262d56c01b not found: ID does not exist" containerID="dbf57086b26e517fa9da3dd64a6356e1d905c90726702ec0287a77262d56c01b"
Oct 30 05:10:22.530607 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:22.530566    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbf57086b26e517fa9da3dd64a6356e1d905c90726702ec0287a77262d56c01b"} err="failed to get container status \"dbf57086b26e517fa9da3dd64a6356e1d905c90726702ec0287a77262d56c01b\": rpc error: code = NotFound desc = could not find container \"dbf57086b26e517fa9da3dd64a6356e1d905c90726702ec0287a77262d56c01b\": container with ID starting with dbf57086b26e517fa9da3dd64a6356e1d905c90726702ec0287a77262d56c01b not found: ID does not exist"
Oct 30 05:10:22.562090 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:22.562063    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc"]
Oct 30 05:10:22.573638 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:22.573615    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-ef844616-openshift-default-f57fcfc4f-fzlkc"]
Oct 30 05:10:22.911572 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:22.911539    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9eed8971-bba2-4f8d-9e3c-4f95f426dab9" path="/var/lib/kubelet/pods/9eed8971-bba2-4f8d-9e3c-4f95f426dab9/volumes"
Oct 30 05:10:22.955822 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:22.955792    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:23.063712 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:23.063664    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:23.222621 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:23.222518    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:23.308622 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:23.308584    5898 patch_prober.go:29] interesting pod/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 30 05:10:23.308988 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:23.308879    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254" podUID="e58b1f9b-6669-485e-93a6-6ef4555b1c8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 30 05:10:23.857049 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:23.856998    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:23.972873 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:23.972843    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:24.385926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:24.384494    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:24.401130 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:24.401094    5898 patch_prober.go:29] interesting pod/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 30 05:10:24.401427 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:24.401398    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254" podUID="e58b1f9b-6669-485e-93a6-6ef4555b1c8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 30 05:10:24.599067 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:24.599023    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254"
Oct 30 05:10:25.308013 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:25.307972    5898 patch_prober.go:29] interesting pod/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 30 05:10:25.308417 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:25.308038    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254" podUID="e58b1f9b-6669-485e-93a6-6ef4555b1c8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 30 05:10:25.617143 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:25.617109    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:26.307197 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:26.307158    5898 patch_prober.go:29] interesting pod/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 30 05:10:26.307383 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:26.307214    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254" podUID="e58b1f9b-6669-485e-93a6-6ef4555b1c8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 30 05:10:26.827400 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:10:26.827358    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:10:27.307208 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:27.307120    5898 patch_prober.go:29] interesting pod/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 30 05:10:27.307208 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:27.307174    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254" podUID="e58b1f9b-6669-485e-93a6-6ef4555b1c8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 30 05:10:27.550245 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:27.550206    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:28.312326 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:28.312295    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254"
Oct 30 05:10:28.313203 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:28.313183    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254"
Oct 30 05:10:29.355777 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:29.355737    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:31.008233 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:31.008202    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:31.160176 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:31.160136    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:31.250100 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:31.250067    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:32.690710 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:32.690643    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:33.023091 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:33.023000    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:34.849137 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:34.849101    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:35.053079 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:35.053041    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr"]
Oct 30 05:10:35.053961 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:35.053944    5898 cpu_manager.go:414] "containerMap: removing container" podUID="5036975c-76af-4820-88b6-3c1b53400520" containerName="maas-api"
Oct 30 05:10:35.054027 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:35.053962    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5036975c-76af-4820-88b6-3c1b53400520" containerName="maas-api"
Oct 30 05:10:35.054027 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:35.053978    5898 cpu_manager.go:414] "containerMap: removing container" podUID="736153cf-ff1d-4f3e-b460-a62cd005c988" containerName="istio-proxy"
Oct 30 05:10:35.054027 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:35.053984    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="736153cf-ff1d-4f3e-b460-a62cd005c988" containerName="istio-proxy"
Oct 30 05:10:35.054027 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:35.053999    5898 cpu_manager.go:414] "containerMap: removing container" podUID="7e29899b-8f40-4c21-a63b-2ce27d602dc3" containerName="istio-proxy"
Oct 30 05:10:35.054027 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:35.054004    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e29899b-8f40-4c21-a63b-2ce27d602dc3" containerName="istio-proxy"
Oct 30 05:10:35.054027 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:35.054018    5898 cpu_manager.go:414] "containerMap: removing container" podUID="9eed8971-bba2-4f8d-9e3c-4f95f426dab9" containerName="istio-proxy"
Oct 30 05:10:35.054027 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:35.054023    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9eed8971-bba2-4f8d-9e3c-4f95f426dab9" containerName="istio-proxy"
Oct 30 05:10:35.054247 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:35.054045    5898 cpu_manager.go:414] "containerMap: removing container" podUID="9eed8971-bba2-4f8d-9e3c-4f95f426dab9" containerName="istio-proxy"
Oct 30 05:10:35.054247 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:35.054049    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9eed8971-bba2-4f8d-9e3c-4f95f426dab9" containerName="istio-proxy"
Oct 30 05:10:35.054247 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:35.054169    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="5036975c-76af-4820-88b6-3c1b53400520" containerName="maas-api"
Oct 30 05:10:35.054247 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:35.054181    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="736153cf-ff1d-4f3e-b460-a62cd005c988" containerName="istio-proxy"
Oct 30 05:10:35.054247 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:35.054188    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e29899b-8f40-4c21-a63b-2ce27d602dc3" containerName="istio-proxy"
Oct 30 05:10:35.054247 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:35.054195    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="9eed8971-bba2-4f8d-9e3c-4f95f426dab9" containerName="istio-proxy"
Oct 30 05:10:35.054247 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:35.054204    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="9eed8971-bba2-4f8d-9e3c-4f95f426dab9" containerName="istio-proxy"
Oct 30 05:10:35.059405 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:35.059381    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr"
Oct 30 05:10:35.077786 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:35.077759    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr"]
Oct 30 05:10:35.193243 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:35.193159    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b566bf45-f269-49ea-bec2-dc5339a10db4-istiod-ca-cert\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr\" (UID: \"b566bf45-f269-49ea-bec2-dc5339a10db4\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr"
Oct 30 05:10:35.193406 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:35.193259    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b566bf45-f269-49ea-bec2-dc5339a10db4-istio-envoy\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr\" (UID: \"b566bf45-f269-49ea-bec2-dc5339a10db4\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr"
Oct 30 05:10:35.193406 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:35.193287    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b566bf45-f269-49ea-bec2-dc5339a10db4-istio-podinfo\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr\" (UID: \"b566bf45-f269-49ea-bec2-dc5339a10db4\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr"
Oct 30 05:10:35.193406 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:35.193307    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqvml\" (UniqueName: \"kubernetes.io/projected/b566bf45-f269-49ea-bec2-dc5339a10db4-kube-api-access-hqvml\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr\" (UID: \"b566bf45-f269-49ea-bec2-dc5339a10db4\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr"
Oct 30 05:10:35.193562 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:35.193432    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b566bf45-f269-49ea-bec2-dc5339a10db4-istio-data\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr\" (UID: \"b566bf45-f269-49ea-bec2-dc5339a10db4\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr"
Oct 30 05:10:35.193562 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:35.193474    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b566bf45-f269-49ea-bec2-dc5339a10db4-credential-socket\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr\" (UID: \"b566bf45-f269-49ea-bec2-dc5339a10db4\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr"
Oct 30 05:10:35.193641 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:35.193551    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b566bf45-f269-49ea-bec2-dc5339a10db4-workload-socket\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr\" (UID: \"b566bf45-f269-49ea-bec2-dc5339a10db4\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr"
Oct 30 05:10:35.193687 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:35.193672    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b566bf45-f269-49ea-bec2-dc5339a10db4-workload-certs\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr\" (UID: \"b566bf45-f269-49ea-bec2-dc5339a10db4\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr"
Oct 30 05:10:35.193833 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:35.193818    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b566bf45-f269-49ea-bec2-dc5339a10db4-istio-token\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr\" (UID: \"b566bf45-f269-49ea-bec2-dc5339a10db4\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr"
Oct 30 05:10:35.295155 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:35.295119    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b566bf45-f269-49ea-bec2-dc5339a10db4-istiod-ca-cert\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr\" (UID: \"b566bf45-f269-49ea-bec2-dc5339a10db4\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr"
Oct 30 05:10:35.295358 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:35.295203    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b566bf45-f269-49ea-bec2-dc5339a10db4-istio-envoy\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr\" (UID: \"b566bf45-f269-49ea-bec2-dc5339a10db4\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr"
Oct 30 05:10:35.295358 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:35.295245    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b566bf45-f269-49ea-bec2-dc5339a10db4-istio-podinfo\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr\" (UID: \"b566bf45-f269-49ea-bec2-dc5339a10db4\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr"
Oct 30 05:10:35.295358 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:35.295269    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqvml\" (UniqueName: \"kubernetes.io/projected/b566bf45-f269-49ea-bec2-dc5339a10db4-kube-api-access-hqvml\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr\" (UID: \"b566bf45-f269-49ea-bec2-dc5339a10db4\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr"
Oct 30 05:10:35.295358 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:35.295316    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b566bf45-f269-49ea-bec2-dc5339a10db4-istio-data\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr\" (UID: \"b566bf45-f269-49ea-bec2-dc5339a10db4\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr"
Oct 30 05:10:35.295358 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:35.295344    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b566bf45-f269-49ea-bec2-dc5339a10db4-credential-socket\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr\" (UID: \"b566bf45-f269-49ea-bec2-dc5339a10db4\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr"
Oct 30 05:10:35.295647 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:35.295374    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b566bf45-f269-49ea-bec2-dc5339a10db4-workload-socket\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr\" (UID: \"b566bf45-f269-49ea-bec2-dc5339a10db4\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr"
Oct 30 05:10:35.295647 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:35.295470    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b566bf45-f269-49ea-bec2-dc5339a10db4-workload-certs\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr\" (UID: \"b566bf45-f269-49ea-bec2-dc5339a10db4\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr"
Oct 30 05:10:35.295647 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:35.295612    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b566bf45-f269-49ea-bec2-dc5339a10db4-istio-token\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr\" (UID: \"b566bf45-f269-49ea-bec2-dc5339a10db4\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr"
Oct 30 05:10:35.295799 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:35.295690    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b566bf45-f269-49ea-bec2-dc5339a10db4-credential-socket\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr\" (UID: \"b566bf45-f269-49ea-bec2-dc5339a10db4\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr"
Oct 30 05:10:35.295996 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:35.295963    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b566bf45-f269-49ea-bec2-dc5339a10db4-workload-socket\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr\" (UID: \"b566bf45-f269-49ea-bec2-dc5339a10db4\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr"
Oct 30 05:10:35.296117 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:35.295974    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b566bf45-f269-49ea-bec2-dc5339a10db4-istiod-ca-cert\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr\" (UID: \"b566bf45-f269-49ea-bec2-dc5339a10db4\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr"
Oct 30 05:10:35.296117 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:35.296057    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b566bf45-f269-49ea-bec2-dc5339a10db4-workload-certs\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr\" (UID: \"b566bf45-f269-49ea-bec2-dc5339a10db4\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr"
Oct 30 05:10:35.296201 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:35.296119    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b566bf45-f269-49ea-bec2-dc5339a10db4-istio-data\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr\" (UID: \"b566bf45-f269-49ea-bec2-dc5339a10db4\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr"
Oct 30 05:10:35.298236 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:35.298216    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b566bf45-f269-49ea-bec2-dc5339a10db4-istio-envoy\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr\" (UID: \"b566bf45-f269-49ea-bec2-dc5339a10db4\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr"
Oct 30 05:10:35.298365 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:35.298349    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b566bf45-f269-49ea-bec2-dc5339a10db4-istio-podinfo\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr\" (UID: \"b566bf45-f269-49ea-bec2-dc5339a10db4\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr"
Oct 30 05:10:35.313305 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:35.313284    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b566bf45-f269-49ea-bec2-dc5339a10db4-istio-token\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr\" (UID: \"b566bf45-f269-49ea-bec2-dc5339a10db4\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr"
Oct 30 05:10:35.313516 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:35.313495    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqvml\" (UniqueName: \"kubernetes.io/projected/b566bf45-f269-49ea-bec2-dc5339a10db4-kube-api-access-hqvml\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr\" (UID: \"b566bf45-f269-49ea-bec2-dc5339a10db4\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr"
Oct 30 05:10:35.374131 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:35.374099    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr"
Oct 30 05:10:35.691373 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:35.691337    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr"]
Oct 30 05:10:35.691559 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:10:35.691390    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb566bf45_f269_49ea_bec2_dc5339a10db4.slice/crio-83e325878418b56d009d56e72f5d384a5b0e4fcf3df096302ec3ba5bde3084d2 WatchSource:0}: Error finding container 83e325878418b56d009d56e72f5d384a5b0e4fcf3df096302ec3ba5bde3084d2: Status 404 returned error can't find the container with id 83e325878418b56d009d56e72f5d384a5b0e4fcf3df096302ec3ba5bde3084d2
Oct 30 05:10:35.694377 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:35.694342    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:10:35.694472 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:35.694423    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:10:35.694512 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:35.694467    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:10:36.510631 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:36.510595    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:36.635316 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:36.635280    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr" event={"ID":"b566bf45-f269-49ea-bec2-dc5339a10db4","Type":"ContainerStarted","Data":"d2be178731961bfed67b4790787b0c6dad3bfed60253570b3ec8588c52620182"}
Oct 30 05:10:36.635538 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:36.635344    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr" event={"ID":"b566bf45-f269-49ea-bec2-dc5339a10db4","Type":"ContainerStarted","Data":"83e325878418b56d009d56e72f5d384a5b0e4fcf3df096302ec3ba5bde3084d2"}
Oct 30 05:10:36.704343 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:36.704289    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr" podStartSLOduration=1.7042728249999999 podStartE2EDuration="1.704272825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:10:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:10:36.689597982 +0000 UTC m=+1565.913417909" watchObservedRunningTime="2026-10-30 05:10:36.704272825 +0000 UTC m=+1565.928092726"
Oct 30 05:10:37.400567 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:37.383264    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr"
Oct 30 05:10:37.400567 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:37.399212    5898 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 05:10:37.400567 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:37.399377    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr" podUID="b566bf45-f269-49ea-bec2-dc5339a10db4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 05:10:37.950666 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:37.950559    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-528d67-bfc8bdc7b-2f26h"]
Oct 30 05:10:37.951468 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:37.951431    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-528d67-bfc8bdc7b-2f26h" podUID="1235a781-d216-46c5-b0b2-e39f62ca8c01" containerName="maas-api" containerID="cri-o://84f0b74943f39ae0127055eb165d90c21eeab526c03835acc75018798d3f50fe" gracePeriod=30
Oct 30 05:10:38.022137 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:38.022100    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:38.532618 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:38.500706    5898 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 05:10:38.532981 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:38.532946    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr" podUID="b566bf45-f269-49ea-bec2-dc5339a10db4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 05:10:38.813925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:38.808271    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:38.856643 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:38.856598    5898 generic.go:356] "Generic (PLEG): container finished" podID="1235a781-d216-46c5-b0b2-e39f62ca8c01" containerID="84f0b74943f39ae0127055eb165d90c21eeab526c03835acc75018798d3f50fe" exitCode=0
Oct 30 05:10:38.857887 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:38.857857    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-528d67-bfc8bdc7b-2f26h" event={"ID":"1235a781-d216-46c5-b0b2-e39f62ca8c01","Type":"ContainerDied","Data":"84f0b74943f39ae0127055eb165d90c21eeab526c03835acc75018798d3f50fe"}
Oct 30 05:10:38.861980 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:38.861322    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-528d67-bfc8bdc7b-2f26h"
Oct 30 05:10:38.914486 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:38.913548    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:38.960453 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:38.960412    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1235a781-d216-46c5-b0b2-e39f62ca8c01-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1235a781-d216-46c5-b0b2-e39f62ca8c01-metrics-certs\") pod \"1235a781-d216-46c5-b0b2-e39f62ca8c01\" (UID: \"1235a781-d216-46c5-b0b2-e39f62ca8c01\") "
Oct 30 05:10:38.960453 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:38.960457    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1235a781-d216-46c5-b0b2-e39f62ca8c01-kube-api-access-xpql9\" (UniqueName: \"kubernetes.io/projected/1235a781-d216-46c5-b0b2-e39f62ca8c01-kube-api-access-xpql9\") pod \"1235a781-d216-46c5-b0b2-e39f62ca8c01\" (UID: \"1235a781-d216-46c5-b0b2-e39f62ca8c01\") "
Oct 30 05:10:38.961043 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:38.960488    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1235a781-d216-46c5-b0b2-e39f62ca8c01-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1235a781-d216-46c5-b0b2-e39f62ca8c01-maas-api-tls\") pod \"1235a781-d216-46c5-b0b2-e39f62ca8c01\" (UID: \"1235a781-d216-46c5-b0b2-e39f62ca8c01\") "
Oct 30 05:10:38.961043 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:38.960640    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1235a781-d216-46c5-b0b2-e39f62ca8c01-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1235a781-d216-46c5-b0b2-e39f62ca8c01-openshift-service-ca\") pod \"1235a781-d216-46c5-b0b2-e39f62ca8c01\" (UID: \"1235a781-d216-46c5-b0b2-e39f62ca8c01\") "
Oct 30 05:10:38.961937 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:38.961886    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1235a781-d216-46c5-b0b2-e39f62ca8c01-openshift-service-ca" pod "1235a781-d216-46c5-b0b2-e39f62ca8c01" (UID: "1235a781-d216-46c5-b0b2-e39f62ca8c01"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:10:38.962631 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:38.962611    5898 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1235a781-d216-46c5-b0b2-e39f62ca8c01-openshift-service-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:10:38.963968 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:38.963940    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1235a781-d216-46c5-b0b2-e39f62ca8c01-metrics-certs" pod "1235a781-d216-46c5-b0b2-e39f62ca8c01" (UID: "1235a781-d216-46c5-b0b2-e39f62ca8c01"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:10:38.964103 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:38.964078    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1235a781-d216-46c5-b0b2-e39f62ca8c01-kube-api-access-xpql9" pod "1235a781-d216-46c5-b0b2-e39f62ca8c01" (UID: "1235a781-d216-46c5-b0b2-e39f62ca8c01"). InnerVolumeSpecName "kube-api-access-xpql9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:10:38.964212 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:38.964195    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1235a781-d216-46c5-b0b2-e39f62ca8c01-maas-api-tls" pod "1235a781-d216-46c5-b0b2-e39f62ca8c01" (UID: "1235a781-d216-46c5-b0b2-e39f62ca8c01"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:10:39.011959 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:39.011921    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:39.067205 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:39.067135    5898 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1235a781-d216-46c5-b0b2-e39f62ca8c01-metrics-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:10:39.067205 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:39.067171    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpql9\" (UniqueName: \"kubernetes.io/projected/1235a781-d216-46c5-b0b2-e39f62ca8c01-kube-api-access-xpql9\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:10:39.067205 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:39.067187    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1235a781-d216-46c5-b0b2-e39f62ca8c01-maas-api-tls\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:10:39.399184 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:39.399147    5898 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 05:10:39.399512 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:39.399482    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr" podUID="b566bf45-f269-49ea-bec2-dc5339a10db4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 05:10:39.501263 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:39.501233    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:39.868210 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:39.868174    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-528d67-bfc8bdc7b-2f26h" event={"ID":"1235a781-d216-46c5-b0b2-e39f62ca8c01","Type":"ContainerDied","Data":"df96a6440d387ed8aefa9b680ada820c99630c0b7ce6bd792dd8fc378bef3c4b"}
Oct 30 05:10:39.868381 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:39.868231    5898 scope.go:122] "RemoveContainer" containerID="84f0b74943f39ae0127055eb165d90c21eeab526c03835acc75018798d3f50fe"
Oct 30 05:10:39.868381 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:39.868187    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-528d67-bfc8bdc7b-2f26h"
Oct 30 05:10:39.955458 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:39.955426    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-528d67-bfc8bdc7b-2f26h"]
Oct 30 05:10:39.968600 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:39.968573    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-528d67-bfc8bdc7b-2f26h"]
Oct 30 05:10:40.356635 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:40.356599    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:40.375283 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:40.375253    5898 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 05:10:40.375510 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:40.375484    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr" podUID="b566bf45-f269-49ea-bec2-dc5339a10db4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 05:10:40.749284 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:40.749196    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:40.918950 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:40.918891    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1235a781-d216-46c5-b0b2-e39f62ca8c01" path="/var/lib/kubelet/pods/1235a781-d216-46c5-b0b2-e39f62ca8c01/volumes"
Oct 30 05:10:41.374743 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:41.374707    5898 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 05:10:41.375410 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:41.375367    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr" podUID="b566bf45-f269-49ea-bec2-dc5339a10db4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 05:10:41.473781 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:41.473743    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:41.727827 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:41.727742    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:42.069794 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:42.069706    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:42.326990 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:42.326958    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:42.413200 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:42.413161    5898 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 05:10:42.413765 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:42.413721    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr" podUID="b566bf45-f269-49ea-bec2-dc5339a10db4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 05:10:43.936118 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:43.936058    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:44.177062 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:44.175990    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr"
Oct 30 05:10:44.177062 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:44.176851    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr"
Oct 30 05:10:44.178611 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:44.178149    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr"
Oct 30 05:10:45.901068 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:45.901036    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:46.001305 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:46.001272    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:46.288882 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:46.288784    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:47.297076 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:47.297043    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-c613f5ea/e2e-lifecycle-model-c613f5ea-kserve-5bf54d4c45-n5286"]
Oct 30 05:10:47.344833 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:47.344801    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:47.916749 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:47.916709    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:49.700704 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:49.700670    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:50.096192 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:50.096159    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk"]
Oct 30 05:10:50.097101 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:50.097075    5898 cpu_manager.go:414] "containerMap: removing container" podUID="1235a781-d216-46c5-b0b2-e39f62ca8c01" containerName="maas-api"
Oct 30 05:10:50.097101 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:50.097102    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1235a781-d216-46c5-b0b2-e39f62ca8c01" containerName="maas-api"
Oct 30 05:10:50.097341 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:50.097325    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="1235a781-d216-46c5-b0b2-e39f62ca8c01" containerName="maas-api"
Oct 30 05:10:50.102561 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:50.102501    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk"
Oct 30 05:10:50.138178 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:50.138148    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk"]
Oct 30 05:10:50.178951 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:50.178918    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a7a5302-4950-4318-bcbc-6ed507e3cb41-istiod-ca-cert\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk\" (UID: \"8a7a5302-4950-4318-bcbc-6ed507e3cb41\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk"
Oct 30 05:10:50.179133 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:50.179071    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a7a5302-4950-4318-bcbc-6ed507e3cb41-istio-data\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk\" (UID: \"8a7a5302-4950-4318-bcbc-6ed507e3cb41\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk"
Oct 30 05:10:50.179133 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:50.179118    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a7a5302-4950-4318-bcbc-6ed507e3cb41-istio-envoy\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk\" (UID: \"8a7a5302-4950-4318-bcbc-6ed507e3cb41\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk"
Oct 30 05:10:50.179261 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:50.179246    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a7a5302-4950-4318-bcbc-6ed507e3cb41-workload-certs\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk\" (UID: \"8a7a5302-4950-4318-bcbc-6ed507e3cb41\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk"
Oct 30 05:10:50.179325 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:50.179282    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a7a5302-4950-4318-bcbc-6ed507e3cb41-istio-token\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk\" (UID: \"8a7a5302-4950-4318-bcbc-6ed507e3cb41\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk"
Oct 30 05:10:50.179444 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:50.179427    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g8s8\" (UniqueName: \"kubernetes.io/projected/8a7a5302-4950-4318-bcbc-6ed507e3cb41-kube-api-access-7g8s8\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk\" (UID: \"8a7a5302-4950-4318-bcbc-6ed507e3cb41\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk"
Oct 30 05:10:50.179529 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:50.179466    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a7a5302-4950-4318-bcbc-6ed507e3cb41-credential-socket\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk\" (UID: \"8a7a5302-4950-4318-bcbc-6ed507e3cb41\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk"
Oct 30 05:10:50.179614 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:50.179580    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a7a5302-4950-4318-bcbc-6ed507e3cb41-istio-podinfo\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk\" (UID: \"8a7a5302-4950-4318-bcbc-6ed507e3cb41\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk"
Oct 30 05:10:50.179698 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:50.179678    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a7a5302-4950-4318-bcbc-6ed507e3cb41-workload-socket\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk\" (UID: \"8a7a5302-4950-4318-bcbc-6ed507e3cb41\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk"
Oct 30 05:10:50.281019 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:50.280982    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a7a5302-4950-4318-bcbc-6ed507e3cb41-istio-data\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk\" (UID: \"8a7a5302-4950-4318-bcbc-6ed507e3cb41\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk"
Oct 30 05:10:50.281186 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:50.281034    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a7a5302-4950-4318-bcbc-6ed507e3cb41-istio-envoy\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk\" (UID: \"8a7a5302-4950-4318-bcbc-6ed507e3cb41\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk"
Oct 30 05:10:50.281186 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:50.281097    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a7a5302-4950-4318-bcbc-6ed507e3cb41-workload-certs\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk\" (UID: \"8a7a5302-4950-4318-bcbc-6ed507e3cb41\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk"
Oct 30 05:10:50.281269 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:50.281216    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a7a5302-4950-4318-bcbc-6ed507e3cb41-istio-token\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk\" (UID: \"8a7a5302-4950-4318-bcbc-6ed507e3cb41\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk"
Oct 30 05:10:50.281387 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:50.281367    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a7a5302-4950-4318-bcbc-6ed507e3cb41-istio-data\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk\" (UID: \"8a7a5302-4950-4318-bcbc-6ed507e3cb41\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk"
Oct 30 05:10:50.281430 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:50.281369    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g8s8\" (UniqueName: \"kubernetes.io/projected/8a7a5302-4950-4318-bcbc-6ed507e3cb41-kube-api-access-7g8s8\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk\" (UID: \"8a7a5302-4950-4318-bcbc-6ed507e3cb41\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk"
Oct 30 05:10:50.281430 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:50.281415    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a7a5302-4950-4318-bcbc-6ed507e3cb41-workload-certs\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk\" (UID: \"8a7a5302-4950-4318-bcbc-6ed507e3cb41\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk"
Oct 30 05:10:50.281515 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:50.281429    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a7a5302-4950-4318-bcbc-6ed507e3cb41-credential-socket\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk\" (UID: \"8a7a5302-4950-4318-bcbc-6ed507e3cb41\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk"
Oct 30 05:10:50.281515 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:50.281491    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a7a5302-4950-4318-bcbc-6ed507e3cb41-istio-podinfo\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk\" (UID: \"8a7a5302-4950-4318-bcbc-6ed507e3cb41\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk"
Oct 30 05:10:50.281626 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:50.281549    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a7a5302-4950-4318-bcbc-6ed507e3cb41-workload-socket\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk\" (UID: \"8a7a5302-4950-4318-bcbc-6ed507e3cb41\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk"
Oct 30 05:10:50.281626 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:50.281617    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a7a5302-4950-4318-bcbc-6ed507e3cb41-istiod-ca-cert\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk\" (UID: \"8a7a5302-4950-4318-bcbc-6ed507e3cb41\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk"
Oct 30 05:10:50.281722 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:50.281645    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a7a5302-4950-4318-bcbc-6ed507e3cb41-credential-socket\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk\" (UID: \"8a7a5302-4950-4318-bcbc-6ed507e3cb41\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk"
Oct 30 05:10:50.281883 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:50.281864    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a7a5302-4950-4318-bcbc-6ed507e3cb41-workload-socket\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk\" (UID: \"8a7a5302-4950-4318-bcbc-6ed507e3cb41\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk"
Oct 30 05:10:50.282236 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:50.282216    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a7a5302-4950-4318-bcbc-6ed507e3cb41-istiod-ca-cert\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk\" (UID: \"8a7a5302-4950-4318-bcbc-6ed507e3cb41\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk"
Oct 30 05:10:50.283473 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:50.283450    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a7a5302-4950-4318-bcbc-6ed507e3cb41-istio-envoy\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk\" (UID: \"8a7a5302-4950-4318-bcbc-6ed507e3cb41\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk"
Oct 30 05:10:50.283808 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:50.283790    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a7a5302-4950-4318-bcbc-6ed507e3cb41-istio-podinfo\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk\" (UID: \"8a7a5302-4950-4318-bcbc-6ed507e3cb41\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk"
Oct 30 05:10:50.300637 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:50.300617    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a7a5302-4950-4318-bcbc-6ed507e3cb41-istio-token\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk\" (UID: \"8a7a5302-4950-4318-bcbc-6ed507e3cb41\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk"
Oct 30 05:10:50.300993 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:50.300974    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g8s8\" (UniqueName: \"kubernetes.io/projected/8a7a5302-4950-4318-bcbc-6ed507e3cb41-kube-api-access-7g8s8\") pod \"e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk\" (UID: \"8a7a5302-4950-4318-bcbc-6ed507e3cb41\") " pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk"
Oct 30 05:10:50.420721 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:50.420643    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk"
Oct 30 05:10:50.741465 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:10:50.741428    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a7a5302_4950_4318_bcbc_6ed507e3cb41.slice/crio-c61227387bc69132dc82fc5fb8af0cbf1cc30dd0da23b1288eca60ed3875b0ab WatchSource:0}: Error finding container c61227387bc69132dc82fc5fb8af0cbf1cc30dd0da23b1288eca60ed3875b0ab: Status 404 returned error can't find the container with id c61227387bc69132dc82fc5fb8af0cbf1cc30dd0da23b1288eca60ed3875b0ab
Oct 30 05:10:50.741798 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:50.741726    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk"]
Oct 30 05:10:50.744557 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:50.744522    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:10:50.744636 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:50.744601    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:10:50.744681 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:50.744642    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:10:50.993958 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:50.993833    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk" event={"ID":"8a7a5302-4950-4318-bcbc-6ed507e3cb41","Type":"ContainerStarted","Data":"f4bb6a31989ddaea45df768aaef886082ad5c04db0d06401e0a8812010dcc949"}
Oct 30 05:10:50.993958 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:50.993891    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk" event={"ID":"8a7a5302-4950-4318-bcbc-6ed507e3cb41","Type":"ContainerStarted","Data":"c61227387bc69132dc82fc5fb8af0cbf1cc30dd0da23b1288eca60ed3875b0ab"}
Oct 30 05:10:51.049197 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:51.049141    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk" podStartSLOduration=1.049127792 podStartE2EDuration="1.049127792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:10:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:10:51.034984971 +0000 UTC m=+1580.258804885" watchObservedRunningTime="2026-10-30 05:10:51.049127792 +0000 UTC m=+1580.272947693"
Oct 30 05:10:51.421450 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:51.421409    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk"
Oct 30 05:10:51.423228 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:51.423203    5898 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Oct 30 05:10:51.423291 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:51.423260    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk" podUID="8a7a5302-4950-4318-bcbc-6ed507e3cb41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Oct 30 05:10:51.611372 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:51.611343    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:52.421078 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:52.421032    5898 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Oct 30 05:10:52.421493 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:52.421099    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk" podUID="8a7a5302-4950-4318-bcbc-6ed507e3cb41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Oct 30 05:10:53.332761 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:53.332729    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:53.421092 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:53.421056    5898 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Oct 30 05:10:53.421510 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:53.421109    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk" podUID="8a7a5302-4950-4318-bcbc-6ed507e3cb41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Oct 30 05:10:53.435622 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:53.435590    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:53.570842 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:53.570813    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:54.421658 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:54.421624    5898 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Oct 30 05:10:54.422094 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:54.421678    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk" podUID="8a7a5302-4950-4318-bcbc-6ed507e3cb41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Oct 30 05:10:55.424259 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:55.424219    5898 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Oct 30 05:10:55.424753 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:55.424285    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk" podUID="8a7a5302-4950-4318-bcbc-6ed507e3cb41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Oct 30 05:10:55.803979 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:55.802993    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9" podUID="1647f909-5580-4219-a2e1-30b4cd994379" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:10:56.021019 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:56.019987    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6" podUID="d30a7a86-f1da-451c-81dd-ebdbaeb4f8af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:10:56.091216 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:56.091170    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/distinct-w5-0b21ba-kserve-7674c59666-m2sbd" podUID="24c6b2d4-3377-4079-8273-3f48064c3a98" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:10:56.112345 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:56.112302    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58" podUID="c27f3b21-1447-4af9-8d99-c04fa9b23cda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:10:56.147016 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:56.146973    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5" podUID="ebf8f4a2-8ac6-48ec-bdba-3f6a99653264" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 05:10:56.147742 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:56.147701    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb" podUID="6f16bd6a-a7f0-49bb-844f-57b777bde2ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:10:56.148035 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:56.148008    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx" podUID="d523008f-c740-4a41-89cb-e82e1ffa0094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 05:10:56.148711 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:56.148674    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49" podUID="05eef4eb-f379-4dc7-a947-123bd85a4968" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:10:56.223362 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:56.223323    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6" podUID="47ff272a-ac5c-4f59-bf7b-5c0692d09386" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Oct 30 05:10:56.351143 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:56.351028    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-c613f5ea/e2e-lifecycle-model-c613f5ea-kserve-5bf54d4c45-n5286" podUID="ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:10:56.426779 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:56.426745    5898 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Oct 30 05:10:56.427384 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:56.427355    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk" podUID="8a7a5302-4950-4318-bcbc-6ed507e3cb41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Oct 30 05:10:56.437275 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:56.437241    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g" podUID="70294a85-1050-4632-b3bf-c3e44a4c0b41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:10:56.528080 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:56.528020    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh" podUID="38428a53-04c7-400a-9c78-52407e982478" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 30 05:10:56.805664 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:56.803250    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9" podUID="1647f909-5580-4219-a2e1-30b4cd994379" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:10:56.805664 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:56.803924    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9"
Oct 30 05:10:56.833225 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:56.832166    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849" podUID="a9707f50-d5e1-409f-b8f3-c58629677aaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:10:57.045053 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:10:57.043457    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:10:57.048324 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:57.047421    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6" podUID="d30a7a86-f1da-451c-81dd-ebdbaeb4f8af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:10:57.077057 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:57.077028    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0b21ba/distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6"
Oct 30 05:10:57.193925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:57.156643    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49" podUID="05eef4eb-f379-4dc7-a947-123bd85a4968" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 05:10:57.193925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:57.157035    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5" podUID="ebf8f4a2-8ac6-48ec-bdba-3f6a99653264" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 05:10:57.193925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:57.125527    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/distinct-w5-0b21ba-kserve-7674c59666-m2sbd" podUID="24c6b2d4-3377-4079-8273-3f48064c3a98" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": context deadline exceeded"
Oct 30 05:10:57.193925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:57.157996    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49"
Oct 30 05:10:57.193925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:57.158050    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5"
Oct 30 05:10:57.193925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:57.158085    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0b21ba/distinct-w5-0b21ba-kserve-7674c59666-m2sbd"
Oct 30 05:10:57.201137 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:57.124644    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58" podUID="c27f3b21-1447-4af9-8d99-c04fa9b23cda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:10:57.230830 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:57.215024    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx" podUID="d523008f-c740-4a41-89cb-e82e1ffa0094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:10:57.231139 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:57.215060    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb" podUID="6f16bd6a-a7f0-49bb-844f-57b777bde2ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 05:10:57.231267 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:57.230002    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6" podUID="47ff272a-ac5c-4f59-bf7b-5c0692d09386" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:10:57.231361 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:57.230040    5898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58" podUID="c27f3b21-1447-4af9-8d99-c04fa9b23cda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": context deadline exceeded"
Oct 30 05:10:57.277268 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:57.236571    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx"
Oct 30 05:10:57.277268 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:57.236629    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb"
Oct 30 05:10:57.277268 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:57.236668    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6"
Oct 30 05:10:57.277268 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:57.236703    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58"
Oct 30 05:10:57.358406 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:57.356818    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-c613f5ea/e2e-lifecycle-model-c613f5ea-kserve-5bf54d4c45-n5286" podUID="ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:10:57.474956 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:57.440830    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g" podUID="70294a85-1050-4632-b3bf-c3e44a4c0b41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:10:57.586939 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:57.582800    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-c613f5ea/e2e-lifecycle-model-c613f5ea-kserve-5bf54d4c45-n5286"
Oct 30 05:10:57.586939 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:57.582957    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g"
Oct 30 05:10:57.612716 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:57.612627    5898 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Oct 30 05:10:57.613559 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:57.613520    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk" podUID="8a7a5302-4950-4318-bcbc-6ed507e3cb41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Oct 30 05:10:57.614083 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:57.614060    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247" podUID="1d865d3e-a6e9-407e-844e-cc75fc7f1354" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 05:10:57.883235 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:57.883138    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9" podUID="1647f909-5580-4219-a2e1-30b4cd994379" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 05:10:57.923151 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:57.921999    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849" podUID="a9707f50-d5e1-409f-b8f3-c58629677aaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:10:57.939924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:57.923473    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849"
Oct 30 05:10:57.989927 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:57.977822    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:58.061940 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:58.052457    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6" podUID="d30a7a86-f1da-451c-81dd-ebdbaeb4f8af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:10:58.145983 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:58.140355    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:10:58.161111 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:58.161060    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/distinct-w5-0b21ba-kserve-7674c59666-m2sbd" podUID="24c6b2d4-3377-4079-8273-3f48064c3a98" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:10:58.162696 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:58.162662    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49" podUID="05eef4eb-f379-4dc7-a947-123bd85a4968" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:10:58.168584 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:58.168546    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5" podUID="ebf8f4a2-8ac6-48ec-bdba-3f6a99653264" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 05:10:58.236037 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:58.235994    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58" podUID="c27f3b21-1447-4af9-8d99-c04fa9b23cda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:10:58.236777 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:58.236745    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx" podUID="d523008f-c740-4a41-89cb-e82e1ffa0094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 05:10:58.252969 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:58.237195    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb" podUID="6f16bd6a-a7f0-49bb-844f-57b777bde2ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 05:10:58.334650 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:58.334606    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6" podUID="47ff272a-ac5c-4f59-bf7b-5c0692d09386" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:10:58.361190 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:58.361147    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-c613f5ea/e2e-lifecycle-model-c613f5ea-kserve-5bf54d4c45-n5286" podUID="ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8000/ready\": context deadline exceeded"
Oct 30 05:10:58.447125 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:58.447016    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g" podUID="70294a85-1050-4632-b3bf-c3e44a4c0b41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:10:58.479668 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:58.479635    5898 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Oct 30 05:10:58.480245 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:58.480207    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk" podUID="8a7a5302-4950-4318-bcbc-6ed507e3cb41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Oct 30 05:10:58.572318 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:58.572285    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx"
Oct 30 05:10:58.627223 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:58.627160    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247" podUID="1d865d3e-a6e9-407e-844e-cc75fc7f1354" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:10:58.629004 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:58.627950    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247"
Oct 30 05:10:58.793256 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:58.793225    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9"
Oct 30 05:10:59.001529 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:59.001010    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849" podUID="a9707f50-d5e1-409f-b8f3-c58629677aaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:10:59.060127 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:59.059463    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6" podUID="d30a7a86-f1da-451c-81dd-ebdbaeb4f8af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:10:59.172924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:59.167959    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/distinct-w5-0b21ba-kserve-7674c59666-m2sbd" podUID="24c6b2d4-3377-4079-8273-3f48064c3a98" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:10:59.172924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:59.169037    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5" podUID="ebf8f4a2-8ac6-48ec-bdba-3f6a99653264" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:10:59.177026 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:59.176994    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49"
Oct 30 05:10:59.211423 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:59.211391    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6"
Oct 30 05:10:59.264953 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:59.250283    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58" podUID="c27f3b21-1447-4af9-8d99-c04fa9b23cda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:10:59.264953 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:59.264113    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb" podUID="6f16bd6a-a7f0-49bb-844f-57b777bde2ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 05:10:59.292922 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:59.282031    5898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5" podUID="ebf8f4a2-8ac6-48ec-bdba-3f6a99653264" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:10:59.405473 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:59.405442    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58"
Oct 30 05:10:59.421950 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:59.406633    5898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9" podUID="1647f909-5580-4219-a2e1-30b4cd994379" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:10:59.446282 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:59.431885    5898 patch_prober.go:29] interesting pod/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Oct 30 05:10:59.446282 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:59.436210    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk" podUID="8a7a5302-4950-4318-bcbc-6ed507e3cb41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Oct 30 05:10:59.446282 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:59.432569    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-c613f5ea/e2e-lifecycle-model-c613f5ea-kserve-5bf54d4c45-n5286" podUID="ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:10:59.446282 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:59.443984    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6" podUID="b09804e3-8135-4f6d-ae34-c45d6da9728e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:10:59.452450 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:59.448543    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g" podUID="70294a85-1050-4632-b3bf-c3e44a4c0b41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:10:59.526963 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:59.522271    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh" podUID="38428a53-04c7-400a-9c78-52407e982478" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:10:59.657773 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:59.562880    5898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849" podUID="a9707f50-d5e1-409f-b8f3-c58629677aaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": context deadline exceeded"
Oct 30 05:10:59.657773 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:59.627986    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247" podUID="1d865d3e-a6e9-407e-844e-cc75fc7f1354" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:10:59.657773 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:59.648657    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0b21ba/distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6"
Oct 30 05:10:59.724782 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:59.708539    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849"
Oct 30 05:10:59.731006 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:59.730165    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-c613f5ea/e2e-lifecycle-model-c613f5ea-kserve-5bf54d4c45-n5286"
Oct 30 05:10:59.742801 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:59.742575    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5"
Oct 30 05:10:59.750859 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:59.749930    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0b21ba/distinct-w5-0b21ba-kserve-7674c59666-m2sbd"
Oct 30 05:10:59.763659 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:59.763633    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb"
Oct 30 05:10:59.820612 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:59.817946    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g"
Oct 30 05:10:59.821267 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:10:59.821246    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247"
Oct 30 05:11:00.486161 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:00.486131    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk"
Oct 30 05:11:00.487459 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:00.486681    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk"
Oct 30 05:11:00.487724 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:00.487706    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk"
Oct 30 05:11:00.955638 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:00.955571    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:11:02.320374 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:02.320323    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-c613f5ea/e2e-lifecycle-model-c613f5ea-kserve-5bf54d4c45-n5286" podUID="ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f" containerName="main" containerID="cri-o://163ac35174564bc849bd44b7c142c921a2c8196ea0ba425090f7d507507e9057" gracePeriod=45
Oct 30 05:11:02.338727 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:02.338691    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-c613f5ea/e2e-lifecycle-model-c613f5ea-kserve-5bf54d4c45-n5286" podUID="ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8000/ready\": dial tcp 10.133.0.111:8000: connect: connection refused"
Oct 30 05:11:02.627190 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:02.627166    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c613f5ea/e2e-lifecycle-model-c613f5ea-kserve-5bf54d4c45-n5286"
Oct 30 05:11:02.680772 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:02.680736    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f-model-cache\") pod \"ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f\" (UID: \"ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f\") "
Oct 30 05:11:02.680959 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:02.680811    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f-tls-certs\" (UniqueName: \"kubernetes.io/secret/ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f-tls-certs\") pod \"ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f\" (UID: \"ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f\") "
Oct 30 05:11:02.680959 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:02.680877    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f-dshm\") pod \"ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f\" (UID: \"ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f\") "
Oct 30 05:11:02.681133 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:02.681109    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f-home\" (UniqueName: \"kubernetes.io/empty-dir/ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f-home\") pod \"ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f\" (UID: \"ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f\") "
Oct 30 05:11:02.681330 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:02.681313    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f-tmp-dir\") pod \"ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f\" (UID: \"ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f\") "
Oct 30 05:11:02.681396 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:02.681372    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f-kube-api-access-9vgft\" (UniqueName: \"kubernetes.io/projected/ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f-kube-api-access-9vgft\") pod \"ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f\" (UID: \"ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f\") "
Oct 30 05:11:02.681790 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:02.681758    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f-home" pod "ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f" (UID: "ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:11:02.682256 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:02.682229    5898 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f-home\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:11:02.684755 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:02.684730    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f-tls-certs" pod "ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f" (UID: "ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:11:02.684931 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:02.684866    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f-kube-api-access-9vgft" pod "ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f" (UID: "ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f"). InnerVolumeSpecName "kube-api-access-9vgft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:11:02.685236 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:02.685095    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f-model-cache" pod "ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f" (UID: "ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:11:02.685460 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:02.685408    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f-tmp-dir" pod "ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f" (UID: "ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:11:02.685687 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:02.685652    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f-dshm" pod "ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f" (UID: "ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:11:02.784105 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:02.784029    5898 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f-tmp-dir\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:11:02.784105 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:02.784105    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vgft\" (UniqueName: \"kubernetes.io/projected/ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f-kube-api-access-9vgft\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:11:02.784408 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:02.784116    5898 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f-model-cache\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:11:02.784408 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:02.784125    5898 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f-tls-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:11:02.784408 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:02.784133    5898 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f-dshm\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:11:02.803999 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:02.803957    5898 generic.go:356] "Generic (PLEG): container finished" podID="ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f" containerID="163ac35174564bc849bd44b7c142c921a2c8196ea0ba425090f7d507507e9057" exitCode=0
Oct 30 05:11:02.804152 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:02.804027    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c613f5ea/e2e-lifecycle-model-c613f5ea-kserve-5bf54d4c45-n5286"
Oct 30 05:11:02.804152 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:02.804041    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c613f5ea/e2e-lifecycle-model-c613f5ea-kserve-5bf54d4c45-n5286" event={"ID":"ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f","Type":"ContainerDied","Data":"163ac35174564bc849bd44b7c142c921a2c8196ea0ba425090f7d507507e9057"}
Oct 30 05:11:02.804152 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:02.804092    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c613f5ea/e2e-lifecycle-model-c613f5ea-kserve-5bf54d4c45-n5286" event={"ID":"ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f","Type":"ContainerDied","Data":"a86105a94a7f00fa768021c694fc72f83443982bebad767330aca895dc56087e"}
Oct 30 05:11:02.804152 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:02.804110    5898 scope.go:122] "RemoveContainer" containerID="163ac35174564bc849bd44b7c142c921a2c8196ea0ba425090f7d507507e9057"
Oct 30 05:11:02.815555 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:02.815538    5898 scope.go:122] "RemoveContainer" containerID="163ac35174564bc849bd44b7c142c921a2c8196ea0ba425090f7d507507e9057"
Oct 30 05:11:02.815865 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:11:02.815843    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"163ac35174564bc849bd44b7c142c921a2c8196ea0ba425090f7d507507e9057\": container with ID starting with 163ac35174564bc849bd44b7c142c921a2c8196ea0ba425090f7d507507e9057 not found: ID does not exist" containerID="163ac35174564bc849bd44b7c142c921a2c8196ea0ba425090f7d507507e9057"
Oct 30 05:11:02.815965 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:02.815871    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"163ac35174564bc849bd44b7c142c921a2c8196ea0ba425090f7d507507e9057"} err="failed to get container status \"163ac35174564bc849bd44b7c142c921a2c8196ea0ba425090f7d507507e9057\": rpc error: code = NotFound desc = could not find container \"163ac35174564bc849bd44b7c142c921a2c8196ea0ba425090f7d507507e9057\": container with ID starting with 163ac35174564bc849bd44b7c142c921a2c8196ea0ba425090f7d507507e9057 not found: ID does not exist"
Oct 30 05:11:02.858093 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:02.858011    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-c613f5ea/e2e-lifecycle-model-c613f5ea-kserve-5bf54d4c45-n5286"]
Oct 30 05:11:02.862212 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:02.862184    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-c613f5ea/e2e-lifecycle-model-c613f5ea-kserve-5bf54d4c45-n5286"]
Oct 30 05:11:02.892050 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:02.892026    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f" path="/var/lib/kubelet/pods/ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f/volumes"
Oct 30 05:11:03.252067 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:03.251989    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:11:05.271307 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:05.271273    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:11:07.006573 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:07.006537    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:11:07.098652 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:07.098619    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:11:09.140184 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:09.140152    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:11:10.884072 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:10.883698    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:11:10.990241 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:10.990205    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:11:11.115104 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:11.115069    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:11:12.929617 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:12.929543    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:11:14.897481 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:14.897432    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:11:16.713214 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:16.713177    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:11:17.283110 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:17.283079    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:11:18.397219 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:18.397186    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:11:20.248145 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:20.248110    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:11:21.420266 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:21.420235    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:11:22.654010 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:22.653969    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:11:24.851023 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:24.850985    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:11:25.394541 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:25.394502    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:11:25.584254 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:25.584208    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:11:26.681500 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:26.681470    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:11:26.826710 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:11:26.826669    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:11:28.617007 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:28.616971    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:11:30.472057 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:30.472027    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:11:32.353827 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:32.353741    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:11:32.854151 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:32.854117    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:11:33.004535 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:33.004501    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:11:33.063948 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:33.063887    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-288e01-86b8bc958-dhvhl"]
Oct 30 05:11:33.194916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:33.194812    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k"]
Oct 30 05:11:33.195412 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:33.195343    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k" podUID="536d83f7-b214-4e92-ad4d-5b00c0fef6aa" containerName="istio-proxy" containerID="cri-o://52617c059b3d70542bae1850a8e510c67ead5fb865513c85a52b20466157f671" gracePeriod=30
Oct 30 05:11:33.204249 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:33.204215    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk"]
Oct 30 05:11:33.204457 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:33.204435    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk" podUID="8a7a5302-4950-4318-bcbc-6ed507e3cb41" containerName="istio-proxy" containerID="cri-o://f4bb6a31989ddaea45df768aaef886082ad5c04db0d06401e0a8812010dcc949" gracePeriod=30
Oct 30 05:11:33.231836 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:33.231804    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:11:33.232089 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:33.232064    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr"]
Oct 30 05:11:33.232325 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:33.232297    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr" podUID="b566bf45-f269-49ea-bec2-dc5339a10db4" containerName="istio-proxy" containerID="cri-o://d2be178731961bfed67b4790787b0c6dad3bfed60253570b3ec8588c52620182" gracePeriod=30
Oct 30 05:11:33.364971 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:33.364894    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:11:34.680538 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:34.680501    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:11:36.467743 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:36.467711    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:11:38.064421 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.064384    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-288e01-86b8bc958-dhvhl" podUID="c370781c-0dfa-4699-92ad-808c257a300d" containerName="payload-processing" containerID="cri-o://987b39049437006c4ba864569a3ea2f273d89248030d214b705af63678623e24" gracePeriod=25
Oct 30 05:11:38.176094 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.176055    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:11:38.509587 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.509560    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-288e01-86b8bc958-dhvhl"
Oct 30 05:11:38.572586 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.572552    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk"
Oct 30 05:11:38.636617 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.636587    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr"
Oct 30 05:11:38.656191 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.656159    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k"
Oct 30 05:11:38.669405 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.669376    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a7a5302-4950-4318-bcbc-6ed507e3cb41-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a7a5302-4950-4318-bcbc-6ed507e3cb41-istio-envoy\") pod \"8a7a5302-4950-4318-bcbc-6ed507e3cb41\" (UID: \"8a7a5302-4950-4318-bcbc-6ed507e3cb41\") "
Oct 30 05:11:38.669559 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.669450    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a7a5302-4950-4318-bcbc-6ed507e3cb41-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a7a5302-4950-4318-bcbc-6ed507e3cb41-credential-socket\") pod \"8a7a5302-4950-4318-bcbc-6ed507e3cb41\" (UID: \"8a7a5302-4950-4318-bcbc-6ed507e3cb41\") "
Oct 30 05:11:38.669559 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.669542    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a7a5302-4950-4318-bcbc-6ed507e3cb41-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a7a5302-4950-4318-bcbc-6ed507e3cb41-workload-certs\") pod \"8a7a5302-4950-4318-bcbc-6ed507e3cb41\" (UID: \"8a7a5302-4950-4318-bcbc-6ed507e3cb41\") "
Oct 30 05:11:38.669677 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.669622    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c370781c-0dfa-4699-92ad-808c257a300d-kube-api-access-k7825\" (UniqueName: \"kubernetes.io/projected/c370781c-0dfa-4699-92ad-808c257a300d-kube-api-access-k7825\") pod \"c370781c-0dfa-4699-92ad-808c257a300d\" (UID: \"c370781c-0dfa-4699-92ad-808c257a300d\") "
Oct 30 05:11:38.669736 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.669720    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a7a5302-4950-4318-bcbc-6ed507e3cb41-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a7a5302-4950-4318-bcbc-6ed507e3cb41-istiod-ca-cert\") pod \"8a7a5302-4950-4318-bcbc-6ed507e3cb41\" (UID: \"8a7a5302-4950-4318-bcbc-6ed507e3cb41\") "
Oct 30 05:11:38.669810 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.669791    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a7a5302-4950-4318-bcbc-6ed507e3cb41-istio-token\" (UniqueName: \"kubernetes.io/projected/8a7a5302-4950-4318-bcbc-6ed507e3cb41-istio-token\") pod \"8a7a5302-4950-4318-bcbc-6ed507e3cb41\" (UID: \"8a7a5302-4950-4318-bcbc-6ed507e3cb41\") "
Oct 30 05:11:38.669810 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.669794    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a7a5302-4950-4318-bcbc-6ed507e3cb41-credential-socket" pod "8a7a5302-4950-4318-bcbc-6ed507e3cb41" (UID: "8a7a5302-4950-4318-bcbc-6ed507e3cb41"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:11:38.669933 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.669847    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8a7a5302-4950-4318-bcbc-6ed507e3cb41-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a7a5302-4950-4318-bcbc-6ed507e3cb41-istio-podinfo\") pod \"8a7a5302-4950-4318-bcbc-6ed507e3cb41\" (UID: \"8a7a5302-4950-4318-bcbc-6ed507e3cb41\") "
Oct 30 05:11:38.669933 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.669888    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a7a5302-4950-4318-bcbc-6ed507e3cb41-kube-api-access-7g8s8\" (UniqueName: \"kubernetes.io/projected/8a7a5302-4950-4318-bcbc-6ed507e3cb41-kube-api-access-7g8s8\") pod \"8a7a5302-4950-4318-bcbc-6ed507e3cb41\" (UID: \"8a7a5302-4950-4318-bcbc-6ed507e3cb41\") "
Oct 30 05:11:38.670053 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.669976    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a7a5302-4950-4318-bcbc-6ed507e3cb41-workload-certs" pod "8a7a5302-4950-4318-bcbc-6ed507e3cb41" (UID: "8a7a5302-4950-4318-bcbc-6ed507e3cb41"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:11:38.670053 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.670011    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c370781c-0dfa-4699-92ad-808c257a300d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c370781c-0dfa-4699-92ad-808c257a300d-plugins-config-volume\") pod \"c370781c-0dfa-4699-92ad-808c257a300d\" (UID: \"c370781c-0dfa-4699-92ad-808c257a300d\") "
Oct 30 05:11:38.670174 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.670066    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a7a5302-4950-4318-bcbc-6ed507e3cb41-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a7a5302-4950-4318-bcbc-6ed507e3cb41-istio-data\") pod \"8a7a5302-4950-4318-bcbc-6ed507e3cb41\" (UID: \"8a7a5302-4950-4318-bcbc-6ed507e3cb41\") "
Oct 30 05:11:38.670174 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.670140    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a7a5302-4950-4318-bcbc-6ed507e3cb41-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a7a5302-4950-4318-bcbc-6ed507e3cb41-workload-socket\") pod \"8a7a5302-4950-4318-bcbc-6ed507e3cb41\" (UID: \"8a7a5302-4950-4318-bcbc-6ed507e3cb41\") "
Oct 30 05:11:38.670277 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.670188    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a7a5302-4950-4318-bcbc-6ed507e3cb41-istiod-ca-cert" pod "8a7a5302-4950-4318-bcbc-6ed507e3cb41" (UID: "8a7a5302-4950-4318-bcbc-6ed507e3cb41"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:11:38.670629 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.670584    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c370781c-0dfa-4699-92ad-808c257a300d-plugins-config-volume" pod "c370781c-0dfa-4699-92ad-808c257a300d" (UID: "c370781c-0dfa-4699-92ad-808c257a300d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:11:38.670993 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.670964    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a7a5302-4950-4318-bcbc-6ed507e3cb41-workload-socket" pod "8a7a5302-4950-4318-bcbc-6ed507e3cb41" (UID: "8a7a5302-4950-4318-bcbc-6ed507e3cb41"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:11:38.671073 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.671010    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a7a5302-4950-4318-bcbc-6ed507e3cb41-istio-data" pod "8a7a5302-4950-4318-bcbc-6ed507e3cb41" (UID: "8a7a5302-4950-4318-bcbc-6ed507e3cb41"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:11:38.671227 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.671201    5898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a7a5302-4950-4318-bcbc-6ed507e3cb41-credential-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:11:38.671313 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.671233    5898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a7a5302-4950-4318-bcbc-6ed507e3cb41-workload-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:11:38.671313 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.671249    5898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a7a5302-4950-4318-bcbc-6ed507e3cb41-istiod-ca-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:11:38.671313 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.671263    5898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c370781c-0dfa-4699-92ad-808c257a300d-plugins-config-volume\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:11:38.671313 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.671278    5898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a7a5302-4950-4318-bcbc-6ed507e3cb41-istio-data\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:11:38.671313 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.671292    5898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a7a5302-4950-4318-bcbc-6ed507e3cb41-workload-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:11:38.673198 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.673177    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a7a5302-4950-4318-bcbc-6ed507e3cb41-kube-api-access-7g8s8" pod "8a7a5302-4950-4318-bcbc-6ed507e3cb41" (UID: "8a7a5302-4950-4318-bcbc-6ed507e3cb41"). InnerVolumeSpecName "kube-api-access-7g8s8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:11:38.673664 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.673630    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a7a5302-4950-4318-bcbc-6ed507e3cb41-istio-podinfo" pod "8a7a5302-4950-4318-bcbc-6ed507e3cb41" (UID: "8a7a5302-4950-4318-bcbc-6ed507e3cb41"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:11:38.673739 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.673676    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c370781c-0dfa-4699-92ad-808c257a300d-kube-api-access-k7825" pod "c370781c-0dfa-4699-92ad-808c257a300d" (UID: "c370781c-0dfa-4699-92ad-808c257a300d"). InnerVolumeSpecName "kube-api-access-k7825". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:11:38.675368 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.675326    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a7a5302-4950-4318-bcbc-6ed507e3cb41-istio-token" pod "8a7a5302-4950-4318-bcbc-6ed507e3cb41" (UID: "8a7a5302-4950-4318-bcbc-6ed507e3cb41"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:11:38.675607 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.675577    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a7a5302-4950-4318-bcbc-6ed507e3cb41-istio-envoy" pod "8a7a5302-4950-4318-bcbc-6ed507e3cb41" (UID: "8a7a5302-4950-4318-bcbc-6ed507e3cb41"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:11:38.772079 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.772046    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b566bf45-f269-49ea-bec2-dc5339a10db4-kube-api-access-hqvml\" (UniqueName: \"kubernetes.io/projected/b566bf45-f269-49ea-bec2-dc5339a10db4-kube-api-access-hqvml\") pod \"b566bf45-f269-49ea-bec2-dc5339a10db4\" (UID: \"b566bf45-f269-49ea-bec2-dc5339a10db4\") "
Oct 30 05:11:38.772251 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.772104    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/536d83f7-b214-4e92-ad4d-5b00c0fef6aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/536d83f7-b214-4e92-ad4d-5b00c0fef6aa-workload-socket\") pod \"536d83f7-b214-4e92-ad4d-5b00c0fef6aa\" (UID: \"536d83f7-b214-4e92-ad4d-5b00c0fef6aa\") "
Oct 30 05:11:38.772251 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.772219    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b566bf45-f269-49ea-bec2-dc5339a10db4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b566bf45-f269-49ea-bec2-dc5339a10db4-istio-envoy\") pod \"b566bf45-f269-49ea-bec2-dc5339a10db4\" (UID: \"b566bf45-f269-49ea-bec2-dc5339a10db4\") "
Oct 30 05:11:38.772366 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.772310    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/536d83f7-b214-4e92-ad4d-5b00c0fef6aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/536d83f7-b214-4e92-ad4d-5b00c0fef6aa-istio-data\") pod \"536d83f7-b214-4e92-ad4d-5b00c0fef6aa\" (UID: \"536d83f7-b214-4e92-ad4d-5b00c0fef6aa\") "
Oct 30 05:11:38.772432 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.772388    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/536d83f7-b214-4e92-ad4d-5b00c0fef6aa-workload-socket" pod "536d83f7-b214-4e92-ad4d-5b00c0fef6aa" (UID: "536d83f7-b214-4e92-ad4d-5b00c0fef6aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:11:38.772432 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.772375    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b566bf45-f269-49ea-bec2-dc5339a10db4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b566bf45-f269-49ea-bec2-dc5339a10db4-istio-podinfo\") pod \"b566bf45-f269-49ea-bec2-dc5339a10db4\" (UID: \"b566bf45-f269-49ea-bec2-dc5339a10db4\") "
Oct 30 05:11:38.772576 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.772452    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/536d83f7-b214-4e92-ad4d-5b00c0fef6aa-kube-api-access-h6hgk\" (UniqueName: \"kubernetes.io/projected/536d83f7-b214-4e92-ad4d-5b00c0fef6aa-kube-api-access-h6hgk\") pod \"536d83f7-b214-4e92-ad4d-5b00c0fef6aa\" (UID: \"536d83f7-b214-4e92-ad4d-5b00c0fef6aa\") "
Oct 30 05:11:38.772576 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.772509    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b566bf45-f269-49ea-bec2-dc5339a10db4-istio-token\" (UniqueName: \"kubernetes.io/projected/b566bf45-f269-49ea-bec2-dc5339a10db4-istio-token\") pod \"b566bf45-f269-49ea-bec2-dc5339a10db4\" (UID: \"b566bf45-f269-49ea-bec2-dc5339a10db4\") "
Oct 30 05:11:38.772576 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.772539    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/536d83f7-b214-4e92-ad4d-5b00c0fef6aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/536d83f7-b214-4e92-ad4d-5b00c0fef6aa-istio-podinfo\") pod \"536d83f7-b214-4e92-ad4d-5b00c0fef6aa\" (UID: \"536d83f7-b214-4e92-ad4d-5b00c0fef6aa\") "
Oct 30 05:11:38.772740 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.772603    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/536d83f7-b214-4e92-ad4d-5b00c0fef6aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/536d83f7-b214-4e92-ad4d-5b00c0fef6aa-credential-socket\") pod \"536d83f7-b214-4e92-ad4d-5b00c0fef6aa\" (UID: \"536d83f7-b214-4e92-ad4d-5b00c0fef6aa\") "
Oct 30 05:11:38.772740 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.772601    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/536d83f7-b214-4e92-ad4d-5b00c0fef6aa-istio-data" pod "536d83f7-b214-4e92-ad4d-5b00c0fef6aa" (UID: "536d83f7-b214-4e92-ad4d-5b00c0fef6aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:11:38.773227 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.773179    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/536d83f7-b214-4e92-ad4d-5b00c0fef6aa-istio-token\" (UniqueName: \"kubernetes.io/projected/536d83f7-b214-4e92-ad4d-5b00c0fef6aa-istio-token\") pod \"536d83f7-b214-4e92-ad4d-5b00c0fef6aa\" (UID: \"536d83f7-b214-4e92-ad4d-5b00c0fef6aa\") "
Oct 30 05:11:38.773227 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.773223    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/536d83f7-b214-4e92-ad4d-5b00c0fef6aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/536d83f7-b214-4e92-ad4d-5b00c0fef6aa-istiod-ca-cert\") pod \"536d83f7-b214-4e92-ad4d-5b00c0fef6aa\" (UID: \"536d83f7-b214-4e92-ad4d-5b00c0fef6aa\") "
Oct 30 05:11:38.773413 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.773339    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b566bf45-f269-49ea-bec2-dc5339a10db4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b566bf45-f269-49ea-bec2-dc5339a10db4-workload-socket\") pod \"b566bf45-f269-49ea-bec2-dc5339a10db4\" (UID: \"b566bf45-f269-49ea-bec2-dc5339a10db4\") "
Oct 30 05:11:38.773413 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.773377    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/536d83f7-b214-4e92-ad4d-5b00c0fef6aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/536d83f7-b214-4e92-ad4d-5b00c0fef6aa-workload-certs\") pod \"536d83f7-b214-4e92-ad4d-5b00c0fef6aa\" (UID: \"536d83f7-b214-4e92-ad4d-5b00c0fef6aa\") "
Oct 30 05:11:38.773535 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.773486    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/536d83f7-b214-4e92-ad4d-5b00c0fef6aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/536d83f7-b214-4e92-ad4d-5b00c0fef6aa-istio-envoy\") pod \"536d83f7-b214-4e92-ad4d-5b00c0fef6aa\" (UID: \"536d83f7-b214-4e92-ad4d-5b00c0fef6aa\") "
Oct 30 05:11:38.773595 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.773540    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b566bf45-f269-49ea-bec2-dc5339a10db4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b566bf45-f269-49ea-bec2-dc5339a10db4-istiod-ca-cert\") pod \"b566bf45-f269-49ea-bec2-dc5339a10db4\" (UID: \"b566bf45-f269-49ea-bec2-dc5339a10db4\") "
Oct 30 05:11:38.773595 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.773578    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b566bf45-f269-49ea-bec2-dc5339a10db4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b566bf45-f269-49ea-bec2-dc5339a10db4-workload-certs\") pod \"b566bf45-f269-49ea-bec2-dc5339a10db4\" (UID: \"b566bf45-f269-49ea-bec2-dc5339a10db4\") "
Oct 30 05:11:38.773710 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.773619    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b566bf45-f269-49ea-bec2-dc5339a10db4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b566bf45-f269-49ea-bec2-dc5339a10db4-istio-data\") pod \"b566bf45-f269-49ea-bec2-dc5339a10db4\" (UID: \"b566bf45-f269-49ea-bec2-dc5339a10db4\") "
Oct 30 05:11:38.773710 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.773700    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b566bf45-f269-49ea-bec2-dc5339a10db4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b566bf45-f269-49ea-bec2-dc5339a10db4-credential-socket\") pod \"b566bf45-f269-49ea-bec2-dc5339a10db4\" (UID: \"b566bf45-f269-49ea-bec2-dc5339a10db4\") "
Oct 30 05:11:38.775611 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.774686    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7g8s8\" (UniqueName: \"kubernetes.io/projected/8a7a5302-4950-4318-bcbc-6ed507e3cb41-kube-api-access-7g8s8\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:11:38.775611 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.774715    5898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/536d83f7-b214-4e92-ad4d-5b00c0fef6aa-istio-data\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:11:38.775611 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.774729    5898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a7a5302-4950-4318-bcbc-6ed507e3cb41-istio-envoy\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:11:38.775611 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.774743    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7825\" (UniqueName: \"kubernetes.io/projected/c370781c-0dfa-4699-92ad-808c257a300d-kube-api-access-k7825\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:11:38.775611 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.774757    5898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/536d83f7-b214-4e92-ad4d-5b00c0fef6aa-workload-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:11:38.775611 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.774771    5898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a7a5302-4950-4318-bcbc-6ed507e3cb41-istio-token\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:11:38.775611 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.774784    5898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a7a5302-4950-4318-bcbc-6ed507e3cb41-istio-podinfo\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:11:38.775611 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.775067    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b566bf45-f269-49ea-bec2-dc5339a10db4-credential-socket" pod "b566bf45-f269-49ea-bec2-dc5339a10db4" (UID: "b566bf45-f269-49ea-bec2-dc5339a10db4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:11:38.775611 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.775144    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/536d83f7-b214-4e92-ad4d-5b00c0fef6aa-credential-socket" pod "536d83f7-b214-4e92-ad4d-5b00c0fef6aa" (UID: "536d83f7-b214-4e92-ad4d-5b00c0fef6aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:11:38.775611 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.775173    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b566bf45-f269-49ea-bec2-dc5339a10db4-istio-data" pod "b566bf45-f269-49ea-bec2-dc5339a10db4" (UID: "b566bf45-f269-49ea-bec2-dc5339a10db4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:11:38.775611 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.775218    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b566bf45-f269-49ea-bec2-dc5339a10db4-workload-socket" pod "b566bf45-f269-49ea-bec2-dc5339a10db4" (UID: "b566bf45-f269-49ea-bec2-dc5339a10db4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:11:38.775611 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.775255    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b566bf45-f269-49ea-bec2-dc5339a10db4-workload-certs" pod "b566bf45-f269-49ea-bec2-dc5339a10db4" (UID: "b566bf45-f269-49ea-bec2-dc5339a10db4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:11:38.775611 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.775294    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/536d83f7-b214-4e92-ad4d-5b00c0fef6aa-workload-certs" pod "536d83f7-b214-4e92-ad4d-5b00c0fef6aa" (UID: "536d83f7-b214-4e92-ad4d-5b00c0fef6aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:11:38.775611 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.775436    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b566bf45-f269-49ea-bec2-dc5339a10db4-istiod-ca-cert" pod "b566bf45-f269-49ea-bec2-dc5339a10db4" (UID: "b566bf45-f269-49ea-bec2-dc5339a10db4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:11:38.775611 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.775570    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/536d83f7-b214-4e92-ad4d-5b00c0fef6aa-istiod-ca-cert" pod "536d83f7-b214-4e92-ad4d-5b00c0fef6aa" (UID: "536d83f7-b214-4e92-ad4d-5b00c0fef6aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:11:38.777136 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.777115    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/536d83f7-b214-4e92-ad4d-5b00c0fef6aa-istio-token" pod "536d83f7-b214-4e92-ad4d-5b00c0fef6aa" (UID: "536d83f7-b214-4e92-ad4d-5b00c0fef6aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:11:38.777409 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.777383    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/536d83f7-b214-4e92-ad4d-5b00c0fef6aa-istio-podinfo" pod "536d83f7-b214-4e92-ad4d-5b00c0fef6aa" (UID: "536d83f7-b214-4e92-ad4d-5b00c0fef6aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:11:38.777409 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.777397    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b566bf45-f269-49ea-bec2-dc5339a10db4-istio-podinfo" pod "b566bf45-f269-49ea-bec2-dc5339a10db4" (UID: "b566bf45-f269-49ea-bec2-dc5339a10db4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:11:38.777608 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.777402    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b566bf45-f269-49ea-bec2-dc5339a10db4-istio-envoy" pod "b566bf45-f269-49ea-bec2-dc5339a10db4" (UID: "b566bf45-f269-49ea-bec2-dc5339a10db4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:11:38.777608 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.777486    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b566bf45-f269-49ea-bec2-dc5339a10db4-kube-api-access-hqvml" pod "b566bf45-f269-49ea-bec2-dc5339a10db4" (UID: "b566bf45-f269-49ea-bec2-dc5339a10db4"). InnerVolumeSpecName "kube-api-access-hqvml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:11:38.777608 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.777564    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b566bf45-f269-49ea-bec2-dc5339a10db4-istio-token" pod "b566bf45-f269-49ea-bec2-dc5339a10db4" (UID: "b566bf45-f269-49ea-bec2-dc5339a10db4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:11:38.778562 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.778540    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/536d83f7-b214-4e92-ad4d-5b00c0fef6aa-istio-envoy" pod "536d83f7-b214-4e92-ad4d-5b00c0fef6aa" (UID: "536d83f7-b214-4e92-ad4d-5b00c0fef6aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:11:38.778807 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.778791    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/536d83f7-b214-4e92-ad4d-5b00c0fef6aa-kube-api-access-h6hgk" pod "536d83f7-b214-4e92-ad4d-5b00c0fef6aa" (UID: "536d83f7-b214-4e92-ad4d-5b00c0fef6aa"). InnerVolumeSpecName "kube-api-access-h6hgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:11:38.876553 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.876474    5898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b566bf45-f269-49ea-bec2-dc5339a10db4-istio-token\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:11:38.876553 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.876506    5898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/536d83f7-b214-4e92-ad4d-5b00c0fef6aa-istio-podinfo\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:11:38.876553 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.876520    5898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/536d83f7-b214-4e92-ad4d-5b00c0fef6aa-credential-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:11:38.876553 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.876536    5898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/536d83f7-b214-4e92-ad4d-5b00c0fef6aa-istio-token\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:11:38.876553 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.876550    5898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/536d83f7-b214-4e92-ad4d-5b00c0fef6aa-istiod-ca-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:11:38.876820 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.876563    5898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b566bf45-f269-49ea-bec2-dc5339a10db4-workload-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:11:38.876820 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.876579    5898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/536d83f7-b214-4e92-ad4d-5b00c0fef6aa-workload-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:11:38.876820 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.876592    5898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/536d83f7-b214-4e92-ad4d-5b00c0fef6aa-istio-envoy\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:11:38.876820 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.876606    5898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b566bf45-f269-49ea-bec2-dc5339a10db4-istiod-ca-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:11:38.876820 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.876618    5898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b566bf45-f269-49ea-bec2-dc5339a10db4-workload-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:11:38.876820 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.876631    5898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b566bf45-f269-49ea-bec2-dc5339a10db4-istio-data\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:11:38.876820 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.876643    5898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b566bf45-f269-49ea-bec2-dc5339a10db4-credential-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:11:38.876820 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.876656    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqvml\" (UniqueName: \"kubernetes.io/projected/b566bf45-f269-49ea-bec2-dc5339a10db4-kube-api-access-hqvml\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:11:38.876820 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.876671    5898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b566bf45-f269-49ea-bec2-dc5339a10db4-istio-envoy\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:11:38.876820 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.876683    5898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b566bf45-f269-49ea-bec2-dc5339a10db4-istio-podinfo\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:11:38.876820 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:38.876697    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6hgk\" (UniqueName: \"kubernetes.io/projected/536d83f7-b214-4e92-ad4d-5b00c0fef6aa-kube-api-access-h6hgk\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:11:39.135679 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:39.135602    5898 generic.go:356] "Generic (PLEG): container finished" podID="b566bf45-f269-49ea-bec2-dc5339a10db4" containerID="d2be178731961bfed67b4790787b0c6dad3bfed60253570b3ec8588c52620182" exitCode=0
Oct 30 05:11:39.135679 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:39.135662    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr"
Oct 30 05:11:39.136111 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:39.135686    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr" event={"ID":"b566bf45-f269-49ea-bec2-dc5339a10db4","Type":"ContainerDied","Data":"d2be178731961bfed67b4790787b0c6dad3bfed60253570b3ec8588c52620182"}
Oct 30 05:11:39.136111 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:39.135769    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr" event={"ID":"b566bf45-f269-49ea-bec2-dc5339a10db4","Type":"ContainerDied","Data":"83e325878418b56d009d56e72f5d384a5b0e4fcf3df096302ec3ba5bde3084d2"}
Oct 30 05:11:39.136111 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:39.135804    5898 scope.go:122] "RemoveContainer" containerID="d2be178731961bfed67b4790787b0c6dad3bfed60253570b3ec8588c52620182"
Oct 30 05:11:39.137254 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:39.137161    5898 generic.go:356] "Generic (PLEG): container finished" podID="c370781c-0dfa-4699-92ad-808c257a300d" containerID="987b39049437006c4ba864569a3ea2f273d89248030d214b705af63678623e24" exitCode=0
Oct 30 05:11:39.137254 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:39.137189    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-288e01-86b8bc958-dhvhl" event={"ID":"c370781c-0dfa-4699-92ad-808c257a300d","Type":"ContainerDied","Data":"987b39049437006c4ba864569a3ea2f273d89248030d214b705af63678623e24"}
Oct 30 05:11:39.137254 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:39.137228    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-288e01-86b8bc958-dhvhl"
Oct 30 05:11:39.137404 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:39.137256    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-288e01-86b8bc958-dhvhl" event={"ID":"c370781c-0dfa-4699-92ad-808c257a300d","Type":"ContainerDied","Data":"0fc7e29ea88bc99c7887188cf187b5c076959abb409d9c3f30f92de0cfc6f57a"}
Oct 30 05:11:39.139042 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:39.139018    5898 generic.go:356] "Generic (PLEG): container finished" podID="536d83f7-b214-4e92-ad4d-5b00c0fef6aa" containerID="52617c059b3d70542bae1850a8e510c67ead5fb865513c85a52b20466157f671" exitCode=0
Oct 30 05:11:39.139191 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:39.139072    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k"
Oct 30 05:11:39.139191 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:39.139096    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k" event={"ID":"536d83f7-b214-4e92-ad4d-5b00c0fef6aa","Type":"ContainerDied","Data":"52617c059b3d70542bae1850a8e510c67ead5fb865513c85a52b20466157f671"}
Oct 30 05:11:39.139191 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:39.139138    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k" event={"ID":"536d83f7-b214-4e92-ad4d-5b00c0fef6aa","Type":"ContainerDied","Data":"41125c0dfefcbded30abed56f5b156a1e9069f221995139351069ad20ded8369"}
Oct 30 05:11:39.141265 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:39.141241    5898 generic.go:356] "Generic (PLEG): container finished" podID="8a7a5302-4950-4318-bcbc-6ed507e3cb41" containerID="f4bb6a31989ddaea45df768aaef886082ad5c04db0d06401e0a8812010dcc949" exitCode=0
Oct 30 05:11:39.141390 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:39.141292    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk"
Oct 30 05:11:39.141390 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:39.141292    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk" event={"ID":"8a7a5302-4950-4318-bcbc-6ed507e3cb41","Type":"ContainerDied","Data":"f4bb6a31989ddaea45df768aaef886082ad5c04db0d06401e0a8812010dcc949"}
Oct 30 05:11:39.141478 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:39.141405    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk" event={"ID":"8a7a5302-4950-4318-bcbc-6ed507e3cb41","Type":"ContainerDied","Data":"c61227387bc69132dc82fc5fb8af0cbf1cc30dd0da23b1288eca60ed3875b0ab"}
Oct 30 05:11:39.150066 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:39.150049    5898 scope.go:122] "RemoveContainer" containerID="d2be178731961bfed67b4790787b0c6dad3bfed60253570b3ec8588c52620182"
Oct 30 05:11:39.150343 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:11:39.150323    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2be178731961bfed67b4790787b0c6dad3bfed60253570b3ec8588c52620182\": container with ID starting with d2be178731961bfed67b4790787b0c6dad3bfed60253570b3ec8588c52620182 not found: ID does not exist" containerID="d2be178731961bfed67b4790787b0c6dad3bfed60253570b3ec8588c52620182"
Oct 30 05:11:39.150442 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:39.150355    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2be178731961bfed67b4790787b0c6dad3bfed60253570b3ec8588c52620182"} err="failed to get container status \"d2be178731961bfed67b4790787b0c6dad3bfed60253570b3ec8588c52620182\": rpc error: code = NotFound desc = could not find container \"d2be178731961bfed67b4790787b0c6dad3bfed60253570b3ec8588c52620182\": container with ID starting with d2be178731961bfed67b4790787b0c6dad3bfed60253570b3ec8588c52620182 not found: ID does not exist"
Oct 30 05:11:39.150442 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:39.150377    5898 scope.go:122] "RemoveContainer" containerID="987b39049437006c4ba864569a3ea2f273d89248030d214b705af63678623e24"
Oct 30 05:11:39.159642 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:39.159615    5898 scope.go:122] "RemoveContainer" containerID="987b39049437006c4ba864569a3ea2f273d89248030d214b705af63678623e24"
Oct 30 05:11:39.159886 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:11:39.159868    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"987b39049437006c4ba864569a3ea2f273d89248030d214b705af63678623e24\": container with ID starting with 987b39049437006c4ba864569a3ea2f273d89248030d214b705af63678623e24 not found: ID does not exist" containerID="987b39049437006c4ba864569a3ea2f273d89248030d214b705af63678623e24"
Oct 30 05:11:39.159970 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:39.159892    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"987b39049437006c4ba864569a3ea2f273d89248030d214b705af63678623e24"} err="failed to get container status \"987b39049437006c4ba864569a3ea2f273d89248030d214b705af63678623e24\": rpc error: code = NotFound desc = could not find container \"987b39049437006c4ba864569a3ea2f273d89248030d214b705af63678623e24\": container with ID starting with 987b39049437006c4ba864569a3ea2f273d89248030d214b705af63678623e24 not found: ID does not exist"
Oct 30 05:11:39.159970 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:39.159945    5898 scope.go:122] "RemoveContainer" containerID="52617c059b3d70542bae1850a8e510c67ead5fb865513c85a52b20466157f671"
Oct 30 05:11:39.169464 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:39.169447    5898 scope.go:122] "RemoveContainer" containerID="52617c059b3d70542bae1850a8e510c67ead5fb865513c85a52b20466157f671"
Oct 30 05:11:39.169746 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:11:39.169721    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52617c059b3d70542bae1850a8e510c67ead5fb865513c85a52b20466157f671\": container with ID starting with 52617c059b3d70542bae1850a8e510c67ead5fb865513c85a52b20466157f671 not found: ID does not exist" containerID="52617c059b3d70542bae1850a8e510c67ead5fb865513c85a52b20466157f671"
Oct 30 05:11:39.169807 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:39.169755    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52617c059b3d70542bae1850a8e510c67ead5fb865513c85a52b20466157f671"} err="failed to get container status \"52617c059b3d70542bae1850a8e510c67ead5fb865513c85a52b20466157f671\": rpc error: code = NotFound desc = could not find container \"52617c059b3d70542bae1850a8e510c67ead5fb865513c85a52b20466157f671\": container with ID starting with 52617c059b3d70542bae1850a8e510c67ead5fb865513c85a52b20466157f671 not found: ID does not exist"
Oct 30 05:11:39.169807 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:39.169777    5898 scope.go:122] "RemoveContainer" containerID="f4bb6a31989ddaea45df768aaef886082ad5c04db0d06401e0a8812010dcc949"
Oct 30 05:11:39.179440 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:39.179416    5898 scope.go:122] "RemoveContainer" containerID="f4bb6a31989ddaea45df768aaef886082ad5c04db0d06401e0a8812010dcc949"
Oct 30 05:11:39.179698 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:11:39.179683    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4bb6a31989ddaea45df768aaef886082ad5c04db0d06401e0a8812010dcc949\": container with ID starting with f4bb6a31989ddaea45df768aaef886082ad5c04db0d06401e0a8812010dcc949 not found: ID does not exist" containerID="f4bb6a31989ddaea45df768aaef886082ad5c04db0d06401e0a8812010dcc949"
Oct 30 05:11:39.179755 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:39.179704    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4bb6a31989ddaea45df768aaef886082ad5c04db0d06401e0a8812010dcc949"} err="failed to get container status \"f4bb6a31989ddaea45df768aaef886082ad5c04db0d06401e0a8812010dcc949\": rpc error: code = NotFound desc = could not find container \"f4bb6a31989ddaea45df768aaef886082ad5c04db0d06401e0a8812010dcc949\": container with ID starting with f4bb6a31989ddaea45df768aaef886082ad5c04db0d06401e0a8812010dcc949 not found: ID does not exist"
Oct 30 05:11:39.198147 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:39.198116    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-288e01-86b8bc958-dhvhl"]
Oct 30 05:11:39.203971 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:39.203946    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-288e01-86b8bc958-dhvhl"]
Oct 30 05:11:39.241523 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:39.241495    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk"]
Oct 30 05:11:39.252073 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:39.252050    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-fjrxk"]
Oct 30 05:11:39.284260 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:39.284225    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr"]
Oct 30 05:11:39.295034 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:39.295006    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-lz8dr"]
Oct 30 05:11:39.327828 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:39.327802    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k"]
Oct 30 05:11:39.331332 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:39.331311    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c613f5ea-openshift-default-55cfff77f7-wd44k"]
Oct 30 05:11:39.881743 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:39.881705    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:11:40.026136 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:40.026106    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:11:40.185347 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:40.185242    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:11:40.688928 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:40.674872    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:11:40.831761 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:40.831728    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:11:40.895028 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:40.894992    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="536d83f7-b214-4e92-ad4d-5b00c0fef6aa" path="/var/lib/kubelet/pods/536d83f7-b214-4e92-ad4d-5b00c0fef6aa/volumes"
Oct 30 05:11:40.895592 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:40.895571    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a7a5302-4950-4318-bcbc-6ed507e3cb41" path="/var/lib/kubelet/pods/8a7a5302-4950-4318-bcbc-6ed507e3cb41/volumes"
Oct 30 05:11:40.896140 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:40.896113    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b566bf45-f269-49ea-bec2-dc5339a10db4" path="/var/lib/kubelet/pods/b566bf45-f269-49ea-bec2-dc5339a10db4/volumes"
Oct 30 05:11:40.896944 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:40.896925    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c370781c-0dfa-4699-92ad-808c257a300d" path="/var/lib/kubelet/pods/c370781c-0dfa-4699-92ad-808c257a300d/volumes"
Oct 30 05:11:40.981531 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:40.981457    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:11:42.293977 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:42.284952    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:11:44.297924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:44.297871    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:11:44.684486 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:44.684446    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:11:44.943798 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:44.943714    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:11:46.495260 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:46.495198    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:11:46.609310 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:46.609237    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:11:47.078497 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:47.078459    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:11:49.143357 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:49.143318    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:11:50.806643 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:50.806608    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:11:51.260689 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:51.260655    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:11:51.408356 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:51.408303    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:11:52.006698 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:52.006659    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:11:54.029695 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:54.029660    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:11:54.137515 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:54.137481    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:11:54.733649 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:54.733610    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:11:55.245185 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:55.245149    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:11:55.387370 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:55.387334    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:11:55.514645 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:55.514556    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:11:56.438034 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:56.437999    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:11:56.819173 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:11:56.819092    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:11:57.300877 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:57.300830    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:11:57.451631 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:57.451572    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:11:59.061061 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:59.061024    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:11:59.350673 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:59.350643    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:11:59.463302 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:11:59.463267    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:01.240658 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:01.240623    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:01.481866 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:01.481836    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:03.558704 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:03.558671    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:04.537298 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:04.537253    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:05.445733 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:05.445693    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:07.229794 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:07.229743    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:09.443718 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:09.443685    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:11.380658 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:11.380620    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:13.187942 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:13.187869    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:15.170708 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:15.170671    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:16.940839 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:16.940804    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:18.416846 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:18.416806    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:19.039401 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:19.039371    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:19.227696 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:19.227660    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:19.344561 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:19.344508    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:19.811113 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:19.811037    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:21.125151 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:21.125116    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:21.802753 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:21.802718    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:22.037999 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:22.037965    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:22.273925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:22.273880    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:22.689940 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:22.689892    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:24.196778 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:24.196745    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:25.191938 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:25.191888    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2"]
Oct 30 05:12:25.192130 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:25.192109    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2" podUID="262d78f1-0e39-4673-a10d-fcf41d4522a4" containerName="istio-proxy" containerID="cri-o://76abacc1cdf1739384e41a9e93c5b59c0eb1bc7b5b35dd46ac44165a254a2624" gracePeriod=30
Oct 30 05:12:26.339717 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:26.339681    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:26.470842 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:26.470812    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:26.624303 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:26.624221    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:26.819325 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:12:26.819287    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:12:28.421671 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:28.421636    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:29.274458 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:29.274422    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:29.409578 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:29.409541    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:29.653790 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:29.653759    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:30.127580 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:30.127543    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:30.256154 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:30.237863    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:30.519921 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:30.519244    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:30.741982 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:30.727518    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:30.945468 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:30.945253    5898 generic.go:356] "Generic (PLEG): container finished" podID="262d78f1-0e39-4673-a10d-fcf41d4522a4" containerID="76abacc1cdf1739384e41a9e93c5b59c0eb1bc7b5b35dd46ac44165a254a2624" exitCode=0
Oct 30 05:12:30.951699 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:30.951655    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2" event={"ID":"262d78f1-0e39-4673-a10d-fcf41d4522a4","Type":"ContainerDied","Data":"76abacc1cdf1739384e41a9e93c5b59c0eb1bc7b5b35dd46ac44165a254a2624"}
Oct 30 05:12:31.000977 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:31.000876    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2"
Oct 30 05:12:31.076986 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:31.076953    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/262d78f1-0e39-4673-a10d-fcf41d4522a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/262d78f1-0e39-4673-a10d-fcf41d4522a4-istiod-ca-cert\") pod \"262d78f1-0e39-4673-a10d-fcf41d4522a4\" (UID: \"262d78f1-0e39-4673-a10d-fcf41d4522a4\") "
Oct 30 05:12:31.077169 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:31.077035    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/262d78f1-0e39-4673-a10d-fcf41d4522a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/262d78f1-0e39-4673-a10d-fcf41d4522a4-workload-socket\") pod \"262d78f1-0e39-4673-a10d-fcf41d4522a4\" (UID: \"262d78f1-0e39-4673-a10d-fcf41d4522a4\") "
Oct 30 05:12:31.077169 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:31.077068    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/262d78f1-0e39-4673-a10d-fcf41d4522a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/262d78f1-0e39-4673-a10d-fcf41d4522a4-workload-certs\") pod \"262d78f1-0e39-4673-a10d-fcf41d4522a4\" (UID: \"262d78f1-0e39-4673-a10d-fcf41d4522a4\") "
Oct 30 05:12:31.077169 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:31.077138    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/262d78f1-0e39-4673-a10d-fcf41d4522a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/262d78f1-0e39-4673-a10d-fcf41d4522a4-istio-envoy\") pod \"262d78f1-0e39-4673-a10d-fcf41d4522a4\" (UID: \"262d78f1-0e39-4673-a10d-fcf41d4522a4\") "
Oct 30 05:12:31.077333 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:31.077194    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/262d78f1-0e39-4673-a10d-fcf41d4522a4-kube-api-access-j5rtq\" (UniqueName: \"kubernetes.io/projected/262d78f1-0e39-4673-a10d-fcf41d4522a4-kube-api-access-j5rtq\") pod \"262d78f1-0e39-4673-a10d-fcf41d4522a4\" (UID: \"262d78f1-0e39-4673-a10d-fcf41d4522a4\") "
Oct 30 05:12:31.077386 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:31.077341    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/262d78f1-0e39-4673-a10d-fcf41d4522a4-workload-socket" pod "262d78f1-0e39-4673-a10d-fcf41d4522a4" (UID: "262d78f1-0e39-4673-a10d-fcf41d4522a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:12:31.077386 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:31.077370    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/262d78f1-0e39-4673-a10d-fcf41d4522a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/262d78f1-0e39-4673-a10d-fcf41d4522a4-istio-data\") pod \"262d78f1-0e39-4673-a10d-fcf41d4522a4\" (UID: \"262d78f1-0e39-4673-a10d-fcf41d4522a4\") "
Oct 30 05:12:31.077490 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:31.077368    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/262d78f1-0e39-4673-a10d-fcf41d4522a4-istiod-ca-cert" pod "262d78f1-0e39-4673-a10d-fcf41d4522a4" (UID: "262d78f1-0e39-4673-a10d-fcf41d4522a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:12:31.077490 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:31.077470    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/262d78f1-0e39-4673-a10d-fcf41d4522a4-istio-token\" (UniqueName: \"kubernetes.io/projected/262d78f1-0e39-4673-a10d-fcf41d4522a4-istio-token\") pod \"262d78f1-0e39-4673-a10d-fcf41d4522a4\" (UID: \"262d78f1-0e39-4673-a10d-fcf41d4522a4\") "
Oct 30 05:12:31.077589 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:31.077475    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/262d78f1-0e39-4673-a10d-fcf41d4522a4-workload-certs" pod "262d78f1-0e39-4673-a10d-fcf41d4522a4" (UID: "262d78f1-0e39-4673-a10d-fcf41d4522a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:12:31.077589 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:31.077536    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/262d78f1-0e39-4673-a10d-fcf41d4522a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/262d78f1-0e39-4673-a10d-fcf41d4522a4-istio-podinfo\") pod \"262d78f1-0e39-4673-a10d-fcf41d4522a4\" (UID: \"262d78f1-0e39-4673-a10d-fcf41d4522a4\") "
Oct 30 05:12:31.077703 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:31.077586    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/262d78f1-0e39-4673-a10d-fcf41d4522a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/262d78f1-0e39-4673-a10d-fcf41d4522a4-credential-socket\") pod \"262d78f1-0e39-4673-a10d-fcf41d4522a4\" (UID: \"262d78f1-0e39-4673-a10d-fcf41d4522a4\") "
Oct 30 05:12:31.077703 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:31.077619    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/262d78f1-0e39-4673-a10d-fcf41d4522a4-istio-data" pod "262d78f1-0e39-4673-a10d-fcf41d4522a4" (UID: "262d78f1-0e39-4673-a10d-fcf41d4522a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:12:31.077868 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:31.077844    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/262d78f1-0e39-4673-a10d-fcf41d4522a4-credential-socket" pod "262d78f1-0e39-4673-a10d-fcf41d4522a4" (UID: "262d78f1-0e39-4673-a10d-fcf41d4522a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:12:31.078156 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:31.078132    5898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/262d78f1-0e39-4673-a10d-fcf41d4522a4-workload-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:12:31.078156 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:31.078156    5898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/262d78f1-0e39-4673-a10d-fcf41d4522a4-workload-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:12:31.078354 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:31.078169    5898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/262d78f1-0e39-4673-a10d-fcf41d4522a4-istio-data\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:12:31.078354 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:31.078177    5898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/262d78f1-0e39-4673-a10d-fcf41d4522a4-credential-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:12:31.078354 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:31.078187    5898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/262d78f1-0e39-4673-a10d-fcf41d4522a4-istiod-ca-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:12:31.079757 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:31.079733    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/262d78f1-0e39-4673-a10d-fcf41d4522a4-istio-token" pod "262d78f1-0e39-4673-a10d-fcf41d4522a4" (UID: "262d78f1-0e39-4673-a10d-fcf41d4522a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:12:31.079933 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:31.079875    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/262d78f1-0e39-4673-a10d-fcf41d4522a4-istio-envoy" pod "262d78f1-0e39-4673-a10d-fcf41d4522a4" (UID: "262d78f1-0e39-4673-a10d-fcf41d4522a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:12:31.080045 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:31.079877    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/262d78f1-0e39-4673-a10d-fcf41d4522a4-istio-podinfo" pod "262d78f1-0e39-4673-a10d-fcf41d4522a4" (UID: "262d78f1-0e39-4673-a10d-fcf41d4522a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:12:31.080045 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:31.079998    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/262d78f1-0e39-4673-a10d-fcf41d4522a4-kube-api-access-j5rtq" pod "262d78f1-0e39-4673-a10d-fcf41d4522a4" (UID: "262d78f1-0e39-4673-a10d-fcf41d4522a4"). InnerVolumeSpecName "kube-api-access-j5rtq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:12:31.179583 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:31.179542    5898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/262d78f1-0e39-4673-a10d-fcf41d4522a4-istio-envoy\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:12:31.179583 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:31.179572    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5rtq\" (UniqueName: \"kubernetes.io/projected/262d78f1-0e39-4673-a10d-fcf41d4522a4-kube-api-access-j5rtq\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:12:31.179583 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:31.179592    5898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/262d78f1-0e39-4673-a10d-fcf41d4522a4-istio-token\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:12:31.179872 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:31.179601    5898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/262d78f1-0e39-4673-a10d-fcf41d4522a4-istio-podinfo\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:12:31.377011 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:31.376972    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:31.880149 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:31.880111    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:31.932015 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:31.931961    5898 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 05:12:31.932184 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:31.932040    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2" podUID="262d78f1-0e39-4673-a10d-fcf41d4522a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:12:31.954780 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:31.954752    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2"
Oct 30 05:12:31.954780 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:31.954769    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2" event={"ID":"262d78f1-0e39-4673-a10d-fcf41d4522a4","Type":"ContainerDied","Data":"46a8247ba423e21e194ad22dbd5803af21404ce8dfef5a75dd9aa5e6c1fcb2d4"}
Oct 30 05:12:31.955020 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:31.954829    5898 scope.go:122] "RemoveContainer" containerID="76abacc1cdf1739384e41a9e93c5b59c0eb1bc7b5b35dd46ac44165a254a2624"
Oct 30 05:12:31.980812 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:31.980789    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:32.020961 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:32.020865    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2"]
Oct 30 05:12:32.056923 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:32.054820    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-qfvc2"]
Oct 30 05:12:32.867173 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:32.867139    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:32.895237 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:32.895205    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="262d78f1-0e39-4673-a10d-fcf41d4522a4" path="/var/lib/kubelet/pods/262d78f1-0e39-4673-a10d-fcf41d4522a4/volumes"
Oct 30 05:12:34.770861 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:34.770821    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:35.752818 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:35.752784    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:35.881614 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:35.881581    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:36.099885 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:36.099843    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:36.241851 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:36.241820    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:36.448573 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:36.448479    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:37.932632 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:37.932603    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:38.048087 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:38.048058    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:39.780882 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:39.780849    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:39.887746 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:39.887712    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:40.724624 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:40.724588    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:41.175115 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:41.175070    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:41.322919 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:41.322866    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:42.069989 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:42.069951    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:43.204022 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:43.203988    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:43.332108 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:43.332076    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:43.824507 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:43.824478    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:44.024194 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:44.021683    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:44.138385 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:44.138289    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:44.271137 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:44.271105    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:44.868873 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:44.868835    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:46.361323 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:46.361291    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:48.107451 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:48.107396    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:50.006309 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:50.006270    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:51.722168 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:51.722140    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:52.289640 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:52.289576    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:12:53.515157 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:53.515124    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:54.830282 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:54.830247    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:54.937703 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:54.937654    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:55.775648 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:55.775603    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:56.819652 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:12:56.819614    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:12:57.551106 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:57.551068    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:12:59.414250 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:12:59.414214    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:01.279858 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:01.279821    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:03.077069 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:03.077035    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:04.846653 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:04.846619    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:06.829880 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:06.829842    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:08.617642 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:08.617609    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:08.800077 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:08.800041    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:09.992979 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:09.992938    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr"]
Oct 30 05:13:09.993804 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:09.993750    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr" podUID="3c677544-d5bf-4024-8c23-15360f51b074" containerName="istio-proxy" containerID="cri-o://666ca71863b326852fe3e3e5c461b4b1f1c6e025038dd578b39e6b8c726369ef" gracePeriod=30
Oct 30 05:13:11.079210 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:11.079177    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:11.593403 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:11.593371    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:12.102306 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:12.102274    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:12.637125 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:12.637083    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:12.750742 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:12.750710    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:13.410265 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:13.410233    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:15.223334 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:15.223296    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:15.276891 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:15.276819    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr"
Oct 30 05:13:15.397603 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:15.397565    5898 generic.go:356] "Generic (PLEG): container finished" podID="3c677544-d5bf-4024-8c23-15360f51b074" containerID="666ca71863b326852fe3e3e5c461b4b1f1c6e025038dd578b39e6b8c726369ef" exitCode=0
Oct 30 05:13:15.397788 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:15.397617    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr"
Oct 30 05:13:15.397788 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:15.397618    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr" event={"ID":"3c677544-d5bf-4024-8c23-15360f51b074","Type":"ContainerDied","Data":"666ca71863b326852fe3e3e5c461b4b1f1c6e025038dd578b39e6b8c726369ef"}
Oct 30 05:13:15.397788 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:15.397754    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr" event={"ID":"3c677544-d5bf-4024-8c23-15360f51b074","Type":"ContainerDied","Data":"4b8bbe7690033842a9dafc3d29a66f25ad73a11dd27000d982a2ecfb85a7f69c"}
Oct 30 05:13:15.397997 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:15.397789    5898 scope.go:122] "RemoveContainer" containerID="666ca71863b326852fe3e3e5c461b4b1f1c6e025038dd578b39e6b8c726369ef"
Oct 30 05:13:15.408467 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:15.408447    5898 scope.go:122] "RemoveContainer" containerID="666ca71863b326852fe3e3e5c461b4b1f1c6e025038dd578b39e6b8c726369ef"
Oct 30 05:13:15.408739 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:13:15.408722    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"666ca71863b326852fe3e3e5c461b4b1f1c6e025038dd578b39e6b8c726369ef\": container with ID starting with 666ca71863b326852fe3e3e5c461b4b1f1c6e025038dd578b39e6b8c726369ef not found: ID does not exist" containerID="666ca71863b326852fe3e3e5c461b4b1f1c6e025038dd578b39e6b8c726369ef"
Oct 30 05:13:15.408784 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:15.408748    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"666ca71863b326852fe3e3e5c461b4b1f1c6e025038dd578b39e6b8c726369ef"} err="failed to get container status \"666ca71863b326852fe3e3e5c461b4b1f1c6e025038dd578b39e6b8c726369ef\": rpc error: code = NotFound desc = could not find container \"666ca71863b326852fe3e3e5c461b4b1f1c6e025038dd578b39e6b8c726369ef\": container with ID starting with 666ca71863b326852fe3e3e5c461b4b1f1c6e025038dd578b39e6b8c726369ef not found: ID does not exist"
Oct 30 05:13:15.448855 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:15.448803    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c677544-d5bf-4024-8c23-15360f51b074-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c677544-d5bf-4024-8c23-15360f51b074-credential-socket\") pod \"3c677544-d5bf-4024-8c23-15360f51b074\" (UID: \"3c677544-d5bf-4024-8c23-15360f51b074\") "
Oct 30 05:13:15.449122 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:15.449001    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c677544-d5bf-4024-8c23-15360f51b074-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c677544-d5bf-4024-8c23-15360f51b074-workload-socket\") pod \"3c677544-d5bf-4024-8c23-15360f51b074\" (UID: \"3c677544-d5bf-4024-8c23-15360f51b074\") "
Oct 30 05:13:15.449204 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:15.449111    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c677544-d5bf-4024-8c23-15360f51b074-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c677544-d5bf-4024-8c23-15360f51b074-istiod-ca-cert\") pod \"3c677544-d5bf-4024-8c23-15360f51b074\" (UID: \"3c677544-d5bf-4024-8c23-15360f51b074\") "
Oct 30 05:13:15.449266 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:15.449232    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c677544-d5bf-4024-8c23-15360f51b074-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c677544-d5bf-4024-8c23-15360f51b074-istio-envoy\") pod \"3c677544-d5bf-4024-8c23-15360f51b074\" (UID: \"3c677544-d5bf-4024-8c23-15360f51b074\") "
Oct 30 05:13:15.449374 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:15.449322    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c677544-d5bf-4024-8c23-15360f51b074-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c677544-d5bf-4024-8c23-15360f51b074-istio-data\") pod \"3c677544-d5bf-4024-8c23-15360f51b074\" (UID: \"3c677544-d5bf-4024-8c23-15360f51b074\") "
Oct 30 05:13:15.449448 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:15.449379    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c677544-d5bf-4024-8c23-15360f51b074-istio-token\" (UniqueName: \"kubernetes.io/projected/3c677544-d5bf-4024-8c23-15360f51b074-istio-token\") pod \"3c677544-d5bf-4024-8c23-15360f51b074\" (UID: \"3c677544-d5bf-4024-8c23-15360f51b074\") "
Oct 30 05:13:15.449448 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:15.449431    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c677544-d5bf-4024-8c23-15360f51b074-credential-socket" pod "3c677544-d5bf-4024-8c23-15360f51b074" (UID: "3c677544-d5bf-4024-8c23-15360f51b074"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:13:15.449565 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:15.449465    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c677544-d5bf-4024-8c23-15360f51b074-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c677544-d5bf-4024-8c23-15360f51b074-istio-podinfo\") pod \"3c677544-d5bf-4024-8c23-15360f51b074\" (UID: \"3c677544-d5bf-4024-8c23-15360f51b074\") "
Oct 30 05:13:15.449565 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:15.449551    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c677544-d5bf-4024-8c23-15360f51b074-kube-api-access-jgrs7\" (UniqueName: \"kubernetes.io/projected/3c677544-d5bf-4024-8c23-15360f51b074-kube-api-access-jgrs7\") pod \"3c677544-d5bf-4024-8c23-15360f51b074\" (UID: \"3c677544-d5bf-4024-8c23-15360f51b074\") "
Oct 30 05:13:15.449671 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:15.449598    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c677544-d5bf-4024-8c23-15360f51b074-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c677544-d5bf-4024-8c23-15360f51b074-workload-certs\") pod \"3c677544-d5bf-4024-8c23-15360f51b074\" (UID: \"3c677544-d5bf-4024-8c23-15360f51b074\") "
Oct 30 05:13:15.450182 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:15.450158    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c677544-d5bf-4024-8c23-15360f51b074-workload-socket" pod "3c677544-d5bf-4024-8c23-15360f51b074" (UID: "3c677544-d5bf-4024-8c23-15360f51b074"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:13:15.450477 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:15.450377    5898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c677544-d5bf-4024-8c23-15360f51b074-credential-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:13:15.450477 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:15.450447    5898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c677544-d5bf-4024-8c23-15360f51b074-workload-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:13:15.450477 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:15.450392    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c677544-d5bf-4024-8c23-15360f51b074-workload-certs" pod "3c677544-d5bf-4024-8c23-15360f51b074" (UID: "3c677544-d5bf-4024-8c23-15360f51b074"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:13:15.450774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:15.450603    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c677544-d5bf-4024-8c23-15360f51b074-istio-data" pod "3c677544-d5bf-4024-8c23-15360f51b074" (UID: "3c677544-d5bf-4024-8c23-15360f51b074"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:13:15.450895 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:15.450859    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c677544-d5bf-4024-8c23-15360f51b074-istiod-ca-cert" pod "3c677544-d5bf-4024-8c23-15360f51b074" (UID: "3c677544-d5bf-4024-8c23-15360f51b074"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:13:15.453100 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:15.453073    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c677544-d5bf-4024-8c23-15360f51b074-istio-token" pod "3c677544-d5bf-4024-8c23-15360f51b074" (UID: "3c677544-d5bf-4024-8c23-15360f51b074"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:13:15.453197 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:15.453178    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c677544-d5bf-4024-8c23-15360f51b074-kube-api-access-jgrs7" pod "3c677544-d5bf-4024-8c23-15360f51b074" (UID: "3c677544-d5bf-4024-8c23-15360f51b074"). InnerVolumeSpecName "kube-api-access-jgrs7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:13:15.453295 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:15.453278    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c677544-d5bf-4024-8c23-15360f51b074-istio-envoy" pod "3c677544-d5bf-4024-8c23-15360f51b074" (UID: "3c677544-d5bf-4024-8c23-15360f51b074"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:13:15.453345 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:15.453312    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c677544-d5bf-4024-8c23-15360f51b074-istio-podinfo" pod "3c677544-d5bf-4024-8c23-15360f51b074" (UID: "3c677544-d5bf-4024-8c23-15360f51b074"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:13:15.552480 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:15.552403    5898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c677544-d5bf-4024-8c23-15360f51b074-istiod-ca-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:13:15.552480 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:15.552431    5898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c677544-d5bf-4024-8c23-15360f51b074-istio-envoy\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:13:15.552480 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:15.552441    5898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c677544-d5bf-4024-8c23-15360f51b074-istio-data\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:13:15.552480 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:15.552449    5898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c677544-d5bf-4024-8c23-15360f51b074-istio-token\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:13:15.552480 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:15.552458    5898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c677544-d5bf-4024-8c23-15360f51b074-istio-podinfo\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:13:15.552480 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:15.552466    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgrs7\" (UniqueName: \"kubernetes.io/projected/3c677544-d5bf-4024-8c23-15360f51b074-kube-api-access-jgrs7\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:13:15.552480 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:15.552476    5898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c677544-d5bf-4024-8c23-15360f51b074-workload-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:13:15.745418 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:15.745380    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr"]
Oct 30 05:13:15.757347 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:15.757316    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-q9xbr"]
Oct 30 05:13:16.656388 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:16.656358    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:16.793058 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:16.793023    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:16.894148 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:16.894119    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c677544-d5bf-4024-8c23-15360f51b074" path="/var/lib/kubelet/pods/3c677544-d5bf-4024-8c23-15360f51b074/volumes"
Oct 30 05:13:17.426309 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:17.426280    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:17.627355 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:17.627318    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:18.373401 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:18.373370    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:18.474298 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:18.474262    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:19.531408 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:19.531373    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:19.736542 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:19.736506    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:19.844345 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:19.844312    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:21.506827 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:21.506787    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:23.306235 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:23.306178    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:24.321683 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:24.321649    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr"]
Oct 30 05:13:24.322136 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:24.321873    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr" podUID="d1a5eec3-1d10-4860-bb38-33a092c84327" containerName="istio-proxy" containerID="cri-o://9ecf01636d6da6d91cf8d450bf43203a6b86eb1e842d3c355c4e4e7a9443c123" gracePeriod=30
Oct 30 05:13:24.333196 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:24.333162    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm"]
Oct 30 05:13:24.333399 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:24.333376    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm" podUID="e371f321-3fd2-4ce5-96a5-625388d49ba0" containerName="istio-proxy" containerID="cri-o://75087f6cc857392c37e2f5080f6aa5bcfad125578c6dd4fbec6c06267e518d2b" gracePeriod=30
Oct 30 05:13:24.990954 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:24.988960    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99"]
Oct 30 05:13:24.990954 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:24.989203    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99" podUID="b2421f64-3c01-4dbc-83b9-51d8a29b3e9c" containerName="istio-proxy" containerID="cri-o://ecb89465a459c6cd8b0072840729601554c930ef82c04372e43aaf70d3497dda" gracePeriod=30
Oct 30 05:13:25.016117 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:25.016086    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:26.819263 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:13:26.819223    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:13:26.991234 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:26.991201    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:27.609566 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:27.609535    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:28.102742 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:28.102705    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:28.593176 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:28.593131    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:28.708655 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:28.708569    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:28.824924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:28.822738    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:29.131254 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:29.131190    5898 patch_prober.go:29] interesting pod/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 05:13:29.131730 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:29.131262    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99" podUID="b2421f64-3c01-4dbc-83b9-51d8a29b3e9c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 05:13:29.514003 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:29.513768    5898 generic.go:356] "Generic (PLEG): container finished" podID="e371f321-3fd2-4ce5-96a5-625388d49ba0" containerID="75087f6cc857392c37e2f5080f6aa5bcfad125578c6dd4fbec6c06267e518d2b" exitCode=0
Oct 30 05:13:29.514003 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:29.513890    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm" event={"ID":"e371f321-3fd2-4ce5-96a5-625388d49ba0","Type":"ContainerDied","Data":"75087f6cc857392c37e2f5080f6aa5bcfad125578c6dd4fbec6c06267e518d2b"}
Oct 30 05:13:29.517243 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:29.516784    5898 generic.go:356] "Generic (PLEG): container finished" podID="d1a5eec3-1d10-4860-bb38-33a092c84327" containerID="9ecf01636d6da6d91cf8d450bf43203a6b86eb1e842d3c355c4e4e7a9443c123" exitCode=0
Oct 30 05:13:29.517243 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:29.517138    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr" event={"ID":"d1a5eec3-1d10-4860-bb38-33a092c84327","Type":"ContainerDied","Data":"9ecf01636d6da6d91cf8d450bf43203a6b86eb1e842d3c355c4e4e7a9443c123"}
Oct 30 05:13:29.664780 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:29.664718    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr"
Oct 30 05:13:29.693631 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:29.693604    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm"
Oct 30 05:13:29.827066 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:29.827035    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1a5eec3-1d10-4860-bb38-33a092c84327-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a5eec3-1d10-4860-bb38-33a092c84327-workload-socket\") pod \"d1a5eec3-1d10-4860-bb38-33a092c84327\" (UID: \"d1a5eec3-1d10-4860-bb38-33a092c84327\") "
Oct 30 05:13:29.827254 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:29.827084    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e371f321-3fd2-4ce5-96a5-625388d49ba0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e371f321-3fd2-4ce5-96a5-625388d49ba0-istio-envoy\") pod \"e371f321-3fd2-4ce5-96a5-625388d49ba0\" (UID: \"e371f321-3fd2-4ce5-96a5-625388d49ba0\") "
Oct 30 05:13:29.827254 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:29.827126    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1a5eec3-1d10-4860-bb38-33a092c84327-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1a5eec3-1d10-4860-bb38-33a092c84327-istio-envoy\") pod \"d1a5eec3-1d10-4860-bb38-33a092c84327\" (UID: \"d1a5eec3-1d10-4860-bb38-33a092c84327\") "
Oct 30 05:13:29.827254 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:29.827200    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1a5eec3-1d10-4860-bb38-33a092c84327-istio-token\" (UniqueName: \"kubernetes.io/projected/d1a5eec3-1d10-4860-bb38-33a092c84327-istio-token\") pod \"d1a5eec3-1d10-4860-bb38-33a092c84327\" (UID: \"d1a5eec3-1d10-4860-bb38-33a092c84327\") "
Oct 30 05:13:29.827254 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:29.827243    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e371f321-3fd2-4ce5-96a5-625388d49ba0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e371f321-3fd2-4ce5-96a5-625388d49ba0-workload-socket\") pod \"e371f321-3fd2-4ce5-96a5-625388d49ba0\" (UID: \"e371f321-3fd2-4ce5-96a5-625388d49ba0\") "
Oct 30 05:13:29.827474 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:29.827289    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e371f321-3fd2-4ce5-96a5-625388d49ba0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e371f321-3fd2-4ce5-96a5-625388d49ba0-credential-socket\") pod \"e371f321-3fd2-4ce5-96a5-625388d49ba0\" (UID: \"e371f321-3fd2-4ce5-96a5-625388d49ba0\") "
Oct 30 05:13:29.827474 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:29.827328    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1a5eec3-1d10-4860-bb38-33a092c84327-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1a5eec3-1d10-4860-bb38-33a092c84327-istiod-ca-cert\") pod \"d1a5eec3-1d10-4860-bb38-33a092c84327\" (UID: \"d1a5eec3-1d10-4860-bb38-33a092c84327\") "
Oct 30 05:13:29.827474 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:29.827347    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1a5eec3-1d10-4860-bb38-33a092c84327-workload-socket" pod "d1a5eec3-1d10-4860-bb38-33a092c84327" (UID: "d1a5eec3-1d10-4860-bb38-33a092c84327"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:13:29.827474 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:29.827374    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1a5eec3-1d10-4860-bb38-33a092c84327-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1a5eec3-1d10-4860-bb38-33a092c84327-istio-data\") pod \"d1a5eec3-1d10-4860-bb38-33a092c84327\" (UID: \"d1a5eec3-1d10-4860-bb38-33a092c84327\") "
Oct 30 05:13:29.827474 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:29.827404    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1a5eec3-1d10-4860-bb38-33a092c84327-kube-api-access-tbxsd\" (UniqueName: \"kubernetes.io/projected/d1a5eec3-1d10-4860-bb38-33a092c84327-kube-api-access-tbxsd\") pod \"d1a5eec3-1d10-4860-bb38-33a092c84327\" (UID: \"d1a5eec3-1d10-4860-bb38-33a092c84327\") "
Oct 30 05:13:29.827474 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:29.827433    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e371f321-3fd2-4ce5-96a5-625388d49ba0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e371f321-3fd2-4ce5-96a5-625388d49ba0-istio-data\") pod \"e371f321-3fd2-4ce5-96a5-625388d49ba0\" (UID: \"e371f321-3fd2-4ce5-96a5-625388d49ba0\") "
Oct 30 05:13:29.827812 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:29.827476    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1a5eec3-1d10-4860-bb38-33a092c84327-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a5eec3-1d10-4860-bb38-33a092c84327-credential-socket\") pod \"d1a5eec3-1d10-4860-bb38-33a092c84327\" (UID: \"d1a5eec3-1d10-4860-bb38-33a092c84327\") "
Oct 30 05:13:29.827812 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:29.827522    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e371f321-3fd2-4ce5-96a5-625388d49ba0-istio-token\" (UniqueName: \"kubernetes.io/projected/e371f321-3fd2-4ce5-96a5-625388d49ba0-istio-token\") pod \"e371f321-3fd2-4ce5-96a5-625388d49ba0\" (UID: \"e371f321-3fd2-4ce5-96a5-625388d49ba0\") "
Oct 30 05:13:29.827812 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:29.827549    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e371f321-3fd2-4ce5-96a5-625388d49ba0-kube-api-access-q2ndm\" (UniqueName: \"kubernetes.io/projected/e371f321-3fd2-4ce5-96a5-625388d49ba0-kube-api-access-q2ndm\") pod \"e371f321-3fd2-4ce5-96a5-625388d49ba0\" (UID: \"e371f321-3fd2-4ce5-96a5-625388d49ba0\") "
Oct 30 05:13:29.827812 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:29.827555    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e371f321-3fd2-4ce5-96a5-625388d49ba0-workload-socket" pod "e371f321-3fd2-4ce5-96a5-625388d49ba0" (UID: "e371f321-3fd2-4ce5-96a5-625388d49ba0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:13:29.828080 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:29.827875    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e371f321-3fd2-4ce5-96a5-625388d49ba0-istio-data" pod "e371f321-3fd2-4ce5-96a5-625388d49ba0" (UID: "e371f321-3fd2-4ce5-96a5-625388d49ba0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:13:29.828080 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:29.827985    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e371f321-3fd2-4ce5-96a5-625388d49ba0-credential-socket" pod "e371f321-3fd2-4ce5-96a5-625388d49ba0" (UID: "e371f321-3fd2-4ce5-96a5-625388d49ba0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:13:29.828080 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:29.828072    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1a5eec3-1d10-4860-bb38-33a092c84327-istio-data" pod "d1a5eec3-1d10-4860-bb38-33a092c84327" (UID: "d1a5eec3-1d10-4860-bb38-33a092c84327"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:13:29.828876 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:29.828845    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1a5eec3-1d10-4860-bb38-33a092c84327-credential-socket" pod "d1a5eec3-1d10-4860-bb38-33a092c84327" (UID: "d1a5eec3-1d10-4860-bb38-33a092c84327"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:13:29.829032 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:29.828965    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1a5eec3-1d10-4860-bb38-33a092c84327-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1a5eec3-1d10-4860-bb38-33a092c84327-workload-certs\") pod \"d1a5eec3-1d10-4860-bb38-33a092c84327\" (UID: \"d1a5eec3-1d10-4860-bb38-33a092c84327\") "
Oct 30 05:13:29.829101 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:29.829061    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1a5eec3-1d10-4860-bb38-33a092c84327-istiod-ca-cert" pod "d1a5eec3-1d10-4860-bb38-33a092c84327" (UID: "d1a5eec3-1d10-4860-bb38-33a092c84327"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:13:29.829202 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:29.829180    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e371f321-3fd2-4ce5-96a5-625388d49ba0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e371f321-3fd2-4ce5-96a5-625388d49ba0-workload-certs\") pod \"e371f321-3fd2-4ce5-96a5-625388d49ba0\" (UID: \"e371f321-3fd2-4ce5-96a5-625388d49ba0\") "
Oct 30 05:13:29.829340 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:29.829315    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1a5eec3-1d10-4860-bb38-33a092c84327-workload-certs" pod "d1a5eec3-1d10-4860-bb38-33a092c84327" (UID: "d1a5eec3-1d10-4860-bb38-33a092c84327"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:13:29.829408 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:29.829344    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e371f321-3fd2-4ce5-96a5-625388d49ba0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e371f321-3fd2-4ce5-96a5-625388d49ba0-istio-podinfo\") pod \"e371f321-3fd2-4ce5-96a5-625388d49ba0\" (UID: \"e371f321-3fd2-4ce5-96a5-625388d49ba0\") "
Oct 30 05:13:29.829408 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:29.829379    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d1a5eec3-1d10-4860-bb38-33a092c84327-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1a5eec3-1d10-4860-bb38-33a092c84327-istio-podinfo\") pod \"d1a5eec3-1d10-4860-bb38-33a092c84327\" (UID: \"d1a5eec3-1d10-4860-bb38-33a092c84327\") "
Oct 30 05:13:29.829510 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:29.829447    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e371f321-3fd2-4ce5-96a5-625388d49ba0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e371f321-3fd2-4ce5-96a5-625388d49ba0-istiod-ca-cert\") pod \"e371f321-3fd2-4ce5-96a5-625388d49ba0\" (UID: \"e371f321-3fd2-4ce5-96a5-625388d49ba0\") "
Oct 30 05:13:29.829587 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:29.829523    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e371f321-3fd2-4ce5-96a5-625388d49ba0-workload-certs" pod "e371f321-3fd2-4ce5-96a5-625388d49ba0" (UID: "e371f321-3fd2-4ce5-96a5-625388d49ba0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:13:29.830118 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:29.830093    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e371f321-3fd2-4ce5-96a5-625388d49ba0-istio-envoy" pod "e371f321-3fd2-4ce5-96a5-625388d49ba0" (UID: "e371f321-3fd2-4ce5-96a5-625388d49ba0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:13:29.830403 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:29.830378    5898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a5eec3-1d10-4860-bb38-33a092c84327-workload-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:13:29.830497 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:29.830410    5898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e371f321-3fd2-4ce5-96a5-625388d49ba0-istio-envoy\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:13:29.830497 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:29.830425    5898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e371f321-3fd2-4ce5-96a5-625388d49ba0-workload-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:13:29.830497 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:29.830437    5898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e371f321-3fd2-4ce5-96a5-625388d49ba0-credential-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:13:29.830497 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:29.830451    5898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1a5eec3-1d10-4860-bb38-33a092c84327-istiod-ca-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:13:29.830497 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:29.830454    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e371f321-3fd2-4ce5-96a5-625388d49ba0-kube-api-access-q2ndm" pod "e371f321-3fd2-4ce5-96a5-625388d49ba0" (UID: "e371f321-3fd2-4ce5-96a5-625388d49ba0"). InnerVolumeSpecName "kube-api-access-q2ndm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:13:29.830497 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:29.830464    5898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1a5eec3-1d10-4860-bb38-33a092c84327-istio-data\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:13:29.830823 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:29.830502    5898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e371f321-3fd2-4ce5-96a5-625388d49ba0-istio-data\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:13:29.830823 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:29.830518    5898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a5eec3-1d10-4860-bb38-33a092c84327-credential-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:13:29.830823 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:29.830534    5898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1a5eec3-1d10-4860-bb38-33a092c84327-workload-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:13:29.830823 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:29.830549    5898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e371f321-3fd2-4ce5-96a5-625388d49ba0-workload-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:13:29.831079 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:29.830924    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e371f321-3fd2-4ce5-96a5-625388d49ba0-istiod-ca-cert" pod "e371f321-3fd2-4ce5-96a5-625388d49ba0" (UID: "e371f321-3fd2-4ce5-96a5-625388d49ba0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:13:29.832446 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:29.832120    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e371f321-3fd2-4ce5-96a5-625388d49ba0-istio-podinfo" pod "e371f321-3fd2-4ce5-96a5-625388d49ba0" (UID: "e371f321-3fd2-4ce5-96a5-625388d49ba0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:13:29.832446 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:29.832122    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1a5eec3-1d10-4860-bb38-33a092c84327-istio-token" pod "d1a5eec3-1d10-4860-bb38-33a092c84327" (UID: "d1a5eec3-1d10-4860-bb38-33a092c84327"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:13:29.833050 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:29.833019    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1a5eec3-1d10-4860-bb38-33a092c84327-kube-api-access-tbxsd" pod "d1a5eec3-1d10-4860-bb38-33a092c84327" (UID: "d1a5eec3-1d10-4860-bb38-33a092c84327"). InnerVolumeSpecName "kube-api-access-tbxsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:13:29.833216 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:29.833194    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e371f321-3fd2-4ce5-96a5-625388d49ba0-istio-token" pod "e371f321-3fd2-4ce5-96a5-625388d49ba0" (UID: "e371f321-3fd2-4ce5-96a5-625388d49ba0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:13:29.834211 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:29.834185    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1a5eec3-1d10-4860-bb38-33a092c84327-istio-envoy" pod "d1a5eec3-1d10-4860-bb38-33a092c84327" (UID: "d1a5eec3-1d10-4860-bb38-33a092c84327"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:13:29.835083 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:29.835061    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d1a5eec3-1d10-4860-bb38-33a092c84327-istio-podinfo" pod "d1a5eec3-1d10-4860-bb38-33a092c84327" (UID: "d1a5eec3-1d10-4860-bb38-33a092c84327"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:13:29.932672 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:29.932574    5898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e371f321-3fd2-4ce5-96a5-625388d49ba0-istio-podinfo\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:13:29.932672 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:29.932613    5898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1a5eec3-1d10-4860-bb38-33a092c84327-istio-podinfo\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:13:29.932672 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:29.932628    5898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e371f321-3fd2-4ce5-96a5-625388d49ba0-istiod-ca-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:13:29.932672 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:29.932641    5898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1a5eec3-1d10-4860-bb38-33a092c84327-istio-envoy\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:13:29.932672 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:29.932653    5898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1a5eec3-1d10-4860-bb38-33a092c84327-istio-token\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:13:29.932672 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:29.932666    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbxsd\" (UniqueName: \"kubernetes.io/projected/d1a5eec3-1d10-4860-bb38-33a092c84327-kube-api-access-tbxsd\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:13:29.933065 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:29.932681    5898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e371f321-3fd2-4ce5-96a5-625388d49ba0-istio-token\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:13:29.933065 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:29.932694    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2ndm\" (UniqueName: \"kubernetes.io/projected/e371f321-3fd2-4ce5-96a5-625388d49ba0-kube-api-access-q2ndm\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:13:30.261460 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:30.261427    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99"
Oct 30 05:13:30.339382 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:30.339347    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2421f64-3c01-4dbc-83b9-51d8a29b3e9c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2421f64-3c01-4dbc-83b9-51d8a29b3e9c-istio-envoy\") pod \"b2421f64-3c01-4dbc-83b9-51d8a29b3e9c\" (UID: \"b2421f64-3c01-4dbc-83b9-51d8a29b3e9c\") "
Oct 30 05:13:30.339609 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:30.339406    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2421f64-3c01-4dbc-83b9-51d8a29b3e9c-istio-token\" (UniqueName: \"kubernetes.io/projected/b2421f64-3c01-4dbc-83b9-51d8a29b3e9c-istio-token\") pod \"b2421f64-3c01-4dbc-83b9-51d8a29b3e9c\" (UID: \"b2421f64-3c01-4dbc-83b9-51d8a29b3e9c\") "
Oct 30 05:13:30.339609 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:30.339486    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2421f64-3c01-4dbc-83b9-51d8a29b3e9c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2421f64-3c01-4dbc-83b9-51d8a29b3e9c-credential-socket\") pod \"b2421f64-3c01-4dbc-83b9-51d8a29b3e9c\" (UID: \"b2421f64-3c01-4dbc-83b9-51d8a29b3e9c\") "
Oct 30 05:13:30.339609 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:30.339539    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2421f64-3c01-4dbc-83b9-51d8a29b3e9c-kube-api-access-prgk5\" (UniqueName: \"kubernetes.io/projected/b2421f64-3c01-4dbc-83b9-51d8a29b3e9c-kube-api-access-prgk5\") pod \"b2421f64-3c01-4dbc-83b9-51d8a29b3e9c\" (UID: \"b2421f64-3c01-4dbc-83b9-51d8a29b3e9c\") "
Oct 30 05:13:30.339609 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:30.339578    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2421f64-3c01-4dbc-83b9-51d8a29b3e9c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2421f64-3c01-4dbc-83b9-51d8a29b3e9c-workload-certs\") pod \"b2421f64-3c01-4dbc-83b9-51d8a29b3e9c\" (UID: \"b2421f64-3c01-4dbc-83b9-51d8a29b3e9c\") "
Oct 30 05:13:30.339609 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:30.339625    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b2421f64-3c01-4dbc-83b9-51d8a29b3e9c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2421f64-3c01-4dbc-83b9-51d8a29b3e9c-istio-podinfo\") pod \"b2421f64-3c01-4dbc-83b9-51d8a29b3e9c\" (UID: \"b2421f64-3c01-4dbc-83b9-51d8a29b3e9c\") "
Oct 30 05:13:30.340100 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:30.339655    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2421f64-3c01-4dbc-83b9-51d8a29b3e9c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2421f64-3c01-4dbc-83b9-51d8a29b3e9c-workload-socket\") pod \"b2421f64-3c01-4dbc-83b9-51d8a29b3e9c\" (UID: \"b2421f64-3c01-4dbc-83b9-51d8a29b3e9c\") "
Oct 30 05:13:30.340100 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:30.339690    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2421f64-3c01-4dbc-83b9-51d8a29b3e9c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2421f64-3c01-4dbc-83b9-51d8a29b3e9c-istio-data\") pod \"b2421f64-3c01-4dbc-83b9-51d8a29b3e9c\" (UID: \"b2421f64-3c01-4dbc-83b9-51d8a29b3e9c\") "
Oct 30 05:13:30.340100 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:30.339758    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2421f64-3c01-4dbc-83b9-51d8a29b3e9c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2421f64-3c01-4dbc-83b9-51d8a29b3e9c-istiod-ca-cert\") pod \"b2421f64-3c01-4dbc-83b9-51d8a29b3e9c\" (UID: \"b2421f64-3c01-4dbc-83b9-51d8a29b3e9c\") "
Oct 30 05:13:30.377962 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:30.345389    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2421f64-3c01-4dbc-83b9-51d8a29b3e9c-istio-envoy" pod "b2421f64-3c01-4dbc-83b9-51d8a29b3e9c" (UID: "b2421f64-3c01-4dbc-83b9-51d8a29b3e9c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:13:30.377962 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:30.345666    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2421f64-3c01-4dbc-83b9-51d8a29b3e9c-istio-data" pod "b2421f64-3c01-4dbc-83b9-51d8a29b3e9c" (UID: "b2421f64-3c01-4dbc-83b9-51d8a29b3e9c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:13:30.377962 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:30.345876    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2421f64-3c01-4dbc-83b9-51d8a29b3e9c-credential-socket" pod "b2421f64-3c01-4dbc-83b9-51d8a29b3e9c" (UID: "b2421f64-3c01-4dbc-83b9-51d8a29b3e9c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:13:30.377962 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:30.346191    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2421f64-3c01-4dbc-83b9-51d8a29b3e9c-workload-socket" pod "b2421f64-3c01-4dbc-83b9-51d8a29b3e9c" (UID: "b2421f64-3c01-4dbc-83b9-51d8a29b3e9c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:13:30.377962 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:30.346694    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2421f64-3c01-4dbc-83b9-51d8a29b3e9c-istiod-ca-cert" pod "b2421f64-3c01-4dbc-83b9-51d8a29b3e9c" (UID: "b2421f64-3c01-4dbc-83b9-51d8a29b3e9c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:13:30.377962 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:30.348146    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2421f64-3c01-4dbc-83b9-51d8a29b3e9c-workload-certs" pod "b2421f64-3c01-4dbc-83b9-51d8a29b3e9c" (UID: "b2421f64-3c01-4dbc-83b9-51d8a29b3e9c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:13:30.377962 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:30.376476    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2421f64-3c01-4dbc-83b9-51d8a29b3e9c-istio-token" pod "b2421f64-3c01-4dbc-83b9-51d8a29b3e9c" (UID: "b2421f64-3c01-4dbc-83b9-51d8a29b3e9c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:13:30.378668 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:30.378480    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2421f64-3c01-4dbc-83b9-51d8a29b3e9c-kube-api-access-prgk5" pod "b2421f64-3c01-4dbc-83b9-51d8a29b3e9c" (UID: "b2421f64-3c01-4dbc-83b9-51d8a29b3e9c"). InnerVolumeSpecName "kube-api-access-prgk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:13:30.380174 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:30.380142    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2421f64-3c01-4dbc-83b9-51d8a29b3e9c-istio-podinfo" pod "b2421f64-3c01-4dbc-83b9-51d8a29b3e9c" (UID: "b2421f64-3c01-4dbc-83b9-51d8a29b3e9c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:13:30.448222 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:30.448192    5898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2421f64-3c01-4dbc-83b9-51d8a29b3e9c-istio-envoy\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:13:30.448222 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:30.448220    5898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2421f64-3c01-4dbc-83b9-51d8a29b3e9c-istio-token\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:13:30.448380 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:30.448231    5898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2421f64-3c01-4dbc-83b9-51d8a29b3e9c-credential-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:13:30.448380 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:30.448240    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prgk5\" (UniqueName: \"kubernetes.io/projected/b2421f64-3c01-4dbc-83b9-51d8a29b3e9c-kube-api-access-prgk5\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:13:30.448380 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:30.448250    5898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2421f64-3c01-4dbc-83b9-51d8a29b3e9c-workload-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:13:30.448380 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:30.448258    5898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2421f64-3c01-4dbc-83b9-51d8a29b3e9c-istio-podinfo\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:13:30.448380 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:30.448268    5898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2421f64-3c01-4dbc-83b9-51d8a29b3e9c-workload-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:13:30.448380 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:30.448279    5898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2421f64-3c01-4dbc-83b9-51d8a29b3e9c-istio-data\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:13:30.448380 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:30.448288    5898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2421f64-3c01-4dbc-83b9-51d8a29b3e9c-istiod-ca-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:13:30.531686 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:30.530885    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm"
Oct 30 05:13:30.531686 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:30.531230    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm" event={"ID":"e371f321-3fd2-4ce5-96a5-625388d49ba0","Type":"ContainerDied","Data":"55915a2c3e965e185896feaadbb1592373f201a5340efde0540ff4a49f2828e9"}
Oct 30 05:13:30.531686 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:30.531368    5898 scope.go:122] "RemoveContainer" containerID="75087f6cc857392c37e2f5080f6aa5bcfad125578c6dd4fbec6c06267e518d2b"
Oct 30 05:13:30.533246 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:30.533198    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr" event={"ID":"d1a5eec3-1d10-4860-bb38-33a092c84327","Type":"ContainerDied","Data":"6b3973695c42b741027290d7223d2aab58621b7fed657b7f0a22848e6243f135"}
Oct 30 05:13:30.533246 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:30.533203    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr"
Oct 30 05:13:30.545994 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:30.545963    5898 generic.go:356] "Generic (PLEG): container finished" podID="b2421f64-3c01-4dbc-83b9-51d8a29b3e9c" containerID="ecb89465a459c6cd8b0072840729601554c930ef82c04372e43aaf70d3497dda" exitCode=0
Oct 30 05:13:30.546120 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:30.546029    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99" event={"ID":"b2421f64-3c01-4dbc-83b9-51d8a29b3e9c","Type":"ContainerDied","Data":"ecb89465a459c6cd8b0072840729601554c930ef82c04372e43aaf70d3497dda"}
Oct 30 05:13:30.546120 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:30.546076    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99" event={"ID":"b2421f64-3c01-4dbc-83b9-51d8a29b3e9c","Type":"ContainerDied","Data":"3f102e477b5fcae6a93c9e3ef275633a3ac084660d2f1b2327735b7e1471ecdd"}
Oct 30 05:13:30.546234 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:30.546162    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99"
Oct 30 05:13:30.560942 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:30.560885    5898 scope.go:122] "RemoveContainer" containerID="9ecf01636d6da6d91cf8d450bf43203a6b86eb1e842d3c355c4e4e7a9443c123"
Oct 30 05:13:30.571487 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:30.571463    5898 scope.go:122] "RemoveContainer" containerID="ecb89465a459c6cd8b0072840729601554c930ef82c04372e43aaf70d3497dda"
Oct 30 05:13:30.581570 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:30.581551    5898 scope.go:122] "RemoveContainer" containerID="ecb89465a459c6cd8b0072840729601554c930ef82c04372e43aaf70d3497dda"
Oct 30 05:13:30.581856 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:13:30.581833    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecb89465a459c6cd8b0072840729601554c930ef82c04372e43aaf70d3497dda\": container with ID starting with ecb89465a459c6cd8b0072840729601554c930ef82c04372e43aaf70d3497dda not found: ID does not exist" containerID="ecb89465a459c6cd8b0072840729601554c930ef82c04372e43aaf70d3497dda"
Oct 30 05:13:30.581940 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:30.581865    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecb89465a459c6cd8b0072840729601554c930ef82c04372e43aaf70d3497dda"} err="failed to get container status \"ecb89465a459c6cd8b0072840729601554c930ef82c04372e43aaf70d3497dda\": rpc error: code = NotFound desc = could not find container \"ecb89465a459c6cd8b0072840729601554c930ef82c04372e43aaf70d3497dda\": container with ID starting with ecb89465a459c6cd8b0072840729601554c930ef82c04372e43aaf70d3497dda not found: ID does not exist"
Oct 30 05:13:30.599084 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:30.599058    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm"]
Oct 30 05:13:30.610576 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:30.610553    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-lhrsm"]
Oct 30 05:13:30.632995 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:30.632964    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99"]
Oct 30 05:13:30.637894 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:30.637868    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-s8z99"]
Oct 30 05:13:30.668641 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:30.668615    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr"]
Oct 30 05:13:30.672243 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:30.672219    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-vmprr"]
Oct 30 05:13:30.894406 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:30.894374    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2421f64-3c01-4dbc-83b9-51d8a29b3e9c" path="/var/lib/kubelet/pods/b2421f64-3c01-4dbc-83b9-51d8a29b3e9c/volumes"
Oct 30 05:13:30.894939 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:30.894919    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1a5eec3-1d10-4860-bb38-33a092c84327" path="/var/lib/kubelet/pods/d1a5eec3-1d10-4860-bb38-33a092c84327/volumes"
Oct 30 05:13:30.895415 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:30.895402    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e371f321-3fd2-4ce5-96a5-625388d49ba0" path="/var/lib/kubelet/pods/e371f321-3fd2-4ce5-96a5-625388d49ba0/volumes"
Oct 30 05:13:30.947445 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:30.947411    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:31.590576 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:31.590543    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:32.859715 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:32.859645    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:32.971042 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:32.971006    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:33.093869 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:33.093838    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:34.653237 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:34.653207    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:34.754882 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:34.754852    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:35.182681 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:35.182367    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:36.912039 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:36.912005    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:13:37.370839 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:37.370790    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:39.502923 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:39.499518    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:40.851312 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:40.851281    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:41.283005 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:41.282893    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:43.025713 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:43.025678    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:43.349272 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:43.349231    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:43.458612 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:43.458572    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:44.868879 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:44.868845    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:46.530230 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:46.530190    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:47.861617 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:47.861550    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:47.961492 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:47.961461    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:48.239138 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:48.239047    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:49.727237 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:49.727203    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:50.425731 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:50.425685    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:50.658807 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:50.658740    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:50.752372 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:50.752291    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:51.228305 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:51.228272    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:51.615199 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:51.615161    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv"]
Oct 30 05:13:51.615894 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:51.615871    5898 cpu_manager.go:414] "containerMap: removing container" podUID="b566bf45-f269-49ea-bec2-dc5339a10db4" containerName="istio-proxy"
Oct 30 05:13:51.615894 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:51.615893    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b566bf45-f269-49ea-bec2-dc5339a10db4" containerName="istio-proxy"
Oct 30 05:13:51.616093 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:51.615943    5898 cpu_manager.go:414] "containerMap: removing container" podUID="d1a5eec3-1d10-4860-bb38-33a092c84327" containerName="istio-proxy"
Oct 30 05:13:51.616093 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:51.615952    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1a5eec3-1d10-4860-bb38-33a092c84327" containerName="istio-proxy"
Oct 30 05:13:51.616093 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:51.615966    5898 cpu_manager.go:414] "containerMap: removing container" podUID="ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f" containerName="main"
Oct 30 05:13:51.616093 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:51.615973    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f" containerName="main"
Oct 30 05:13:51.616093 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:51.615996    5898 cpu_manager.go:414] "containerMap: removing container" podUID="536d83f7-b214-4e92-ad4d-5b00c0fef6aa" containerName="istio-proxy"
Oct 30 05:13:51.616093 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:51.616003    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="536d83f7-b214-4e92-ad4d-5b00c0fef6aa" containerName="istio-proxy"
Oct 30 05:13:51.616093 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:51.616028    5898 cpu_manager.go:414] "containerMap: removing container" podUID="3c677544-d5bf-4024-8c23-15360f51b074" containerName="istio-proxy"
Oct 30 05:13:51.616093 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:51.616037    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c677544-d5bf-4024-8c23-15360f51b074" containerName="istio-proxy"
Oct 30 05:13:51.616093 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:51.616048    5898 cpu_manager.go:414] "containerMap: removing container" podUID="8a7a5302-4950-4318-bcbc-6ed507e3cb41" containerName="istio-proxy"
Oct 30 05:13:51.616093 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:51.616056    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a7a5302-4950-4318-bcbc-6ed507e3cb41" containerName="istio-proxy"
Oct 30 05:13:51.616093 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:51.616086    5898 cpu_manager.go:414] "containerMap: removing container" podUID="c370781c-0dfa-4699-92ad-808c257a300d" containerName="payload-processing"
Oct 30 05:13:51.616093 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:51.616093    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c370781c-0dfa-4699-92ad-808c257a300d" containerName="payload-processing"
Oct 30 05:13:51.616683 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:51.616111    5898 cpu_manager.go:414] "containerMap: removing container" podUID="b2421f64-3c01-4dbc-83b9-51d8a29b3e9c" containerName="istio-proxy"
Oct 30 05:13:51.616683 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:51.616118    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2421f64-3c01-4dbc-83b9-51d8a29b3e9c" containerName="istio-proxy"
Oct 30 05:13:51.616683 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:51.616129    5898 cpu_manager.go:414] "containerMap: removing container" podUID="262d78f1-0e39-4673-a10d-fcf41d4522a4" containerName="istio-proxy"
Oct 30 05:13:51.616683 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:51.616136    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="262d78f1-0e39-4673-a10d-fcf41d4522a4" containerName="istio-proxy"
Oct 30 05:13:51.616683 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:51.616144    5898 cpu_manager.go:414] "containerMap: removing container" podUID="e371f321-3fd2-4ce5-96a5-625388d49ba0" containerName="istio-proxy"
Oct 30 05:13:51.616683 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:51.616151    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e371f321-3fd2-4ce5-96a5-625388d49ba0" containerName="istio-proxy"
Oct 30 05:13:51.616683 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:51.616307    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1a5eec3-1d10-4860-bb38-33a092c84327" containerName="istio-proxy"
Oct 30 05:13:51.616683 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:51.616321    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec2a0e9c-64ab-4b68-8caf-20c9f8f9126f" containerName="main"
Oct 30 05:13:51.616683 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:51.616334    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="536d83f7-b214-4e92-ad4d-5b00c0fef6aa" containerName="istio-proxy"
Oct 30 05:13:51.616683 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:51.616347    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c677544-d5bf-4024-8c23-15360f51b074" containerName="istio-proxy"
Oct 30 05:13:51.616683 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:51.616357    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a7a5302-4950-4318-bcbc-6ed507e3cb41" containerName="istio-proxy"
Oct 30 05:13:51.616683 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:51.616367    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="c370781c-0dfa-4699-92ad-808c257a300d" containerName="payload-processing"
Oct 30 05:13:51.616683 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:51.616377    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2421f64-3c01-4dbc-83b9-51d8a29b3e9c" containerName="istio-proxy"
Oct 30 05:13:51.616683 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:51.616385    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="262d78f1-0e39-4673-a10d-fcf41d4522a4" containerName="istio-proxy"
Oct 30 05:13:51.616683 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:51.616395    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="e371f321-3fd2-4ce5-96a5-625388d49ba0" containerName="istio-proxy"
Oct 30 05:13:51.616683 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:51.616403    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="b566bf45-f269-49ea-bec2-dc5339a10db4" containerName="istio-proxy"
Oct 30 05:13:51.621981 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:51.621960    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv"
Oct 30 05:13:51.642562 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:51.642525    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv"]
Oct 30 05:13:51.708627 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:51.708593    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9bee146-e705-478f-abda-fa3111705a69-istio-envoy\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv\" (UID: \"c9bee146-e705-478f-abda-fa3111705a69\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv"
Oct 30 05:13:51.708779 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:51.708643    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9bee146-e705-478f-abda-fa3111705a69-istio-podinfo\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv\" (UID: \"c9bee146-e705-478f-abda-fa3111705a69\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv"
Oct 30 05:13:51.708779 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:51.708719    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bee146-e705-478f-abda-fa3111705a69-workload-socket\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv\" (UID: \"c9bee146-e705-478f-abda-fa3111705a69\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv"
Oct 30 05:13:51.708779 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:51.708765    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bee146-e705-478f-abda-fa3111705a69-credential-socket\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv\" (UID: \"c9bee146-e705-478f-abda-fa3111705a69\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv"
Oct 30 05:13:51.708893 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:51.708784    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9bee146-e705-478f-abda-fa3111705a69-istio-token\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv\" (UID: \"c9bee146-e705-478f-abda-fa3111705a69\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv"
Oct 30 05:13:51.708971 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:51.708938    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9bee146-e705-478f-abda-fa3111705a69-workload-certs\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv\" (UID: \"c9bee146-e705-478f-abda-fa3111705a69\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv"
Oct 30 05:13:51.709010 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:51.708973    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9bee146-e705-478f-abda-fa3111705a69-istiod-ca-cert\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv\" (UID: \"c9bee146-e705-478f-abda-fa3111705a69\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv"
Oct 30 05:13:51.709010 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:51.708999    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-659kd\" (UniqueName: \"kubernetes.io/projected/c9bee146-e705-478f-abda-fa3111705a69-kube-api-access-659kd\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv\" (UID: \"c9bee146-e705-478f-abda-fa3111705a69\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv"
Oct 30 05:13:51.709097 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:51.709082    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9bee146-e705-478f-abda-fa3111705a69-istio-data\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv\" (UID: \"c9bee146-e705-478f-abda-fa3111705a69\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv"
Oct 30 05:13:51.727720 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:51.727697    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-1f8eb4-openshift-default-dockercfg-kktfc\""
Oct 30 05:13:51.810183 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:51.810144    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9bee146-e705-478f-abda-fa3111705a69-istio-podinfo\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv\" (UID: \"c9bee146-e705-478f-abda-fa3111705a69\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv"
Oct 30 05:13:51.810183 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:51.810196    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bee146-e705-478f-abda-fa3111705a69-workload-socket\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv\" (UID: \"c9bee146-e705-478f-abda-fa3111705a69\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv"
Oct 30 05:13:51.810773 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:51.810230    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bee146-e705-478f-abda-fa3111705a69-credential-socket\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv\" (UID: \"c9bee146-e705-478f-abda-fa3111705a69\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv"
Oct 30 05:13:51.810773 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:51.810247    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9bee146-e705-478f-abda-fa3111705a69-istio-token\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv\" (UID: \"c9bee146-e705-478f-abda-fa3111705a69\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv"
Oct 30 05:13:51.810773 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:51.810315    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9bee146-e705-478f-abda-fa3111705a69-workload-certs\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv\" (UID: \"c9bee146-e705-478f-abda-fa3111705a69\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv"
Oct 30 05:13:51.810773 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:51.810353    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9bee146-e705-478f-abda-fa3111705a69-istiod-ca-cert\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv\" (UID: \"c9bee146-e705-478f-abda-fa3111705a69\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv"
Oct 30 05:13:51.810773 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:51.810388    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-659kd\" (UniqueName: \"kubernetes.io/projected/c9bee146-e705-478f-abda-fa3111705a69-kube-api-access-659kd\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv\" (UID: \"c9bee146-e705-478f-abda-fa3111705a69\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv"
Oct 30 05:13:51.810773 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:51.810442    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9bee146-e705-478f-abda-fa3111705a69-istio-data\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv\" (UID: \"c9bee146-e705-478f-abda-fa3111705a69\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv"
Oct 30 05:13:51.810773 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:51.810640    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9bee146-e705-478f-abda-fa3111705a69-istio-envoy\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv\" (UID: \"c9bee146-e705-478f-abda-fa3111705a69\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv"
Oct 30 05:13:51.810773 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:51.810652    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bee146-e705-478f-abda-fa3111705a69-workload-socket\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv\" (UID: \"c9bee146-e705-478f-abda-fa3111705a69\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv"
Oct 30 05:13:51.810773 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:51.810701    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bee146-e705-478f-abda-fa3111705a69-credential-socket\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv\" (UID: \"c9bee146-e705-478f-abda-fa3111705a69\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv"
Oct 30 05:13:51.811304 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:51.810936    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9bee146-e705-478f-abda-fa3111705a69-istio-data\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv\" (UID: \"c9bee146-e705-478f-abda-fa3111705a69\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv"
Oct 30 05:13:51.811304 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:51.810960    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9bee146-e705-478f-abda-fa3111705a69-workload-certs\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv\" (UID: \"c9bee146-e705-478f-abda-fa3111705a69\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv"
Oct 30 05:13:51.811304 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:51.811106    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9bee146-e705-478f-abda-fa3111705a69-istiod-ca-cert\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv\" (UID: \"c9bee146-e705-478f-abda-fa3111705a69\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv"
Oct 30 05:13:51.812788 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:51.812768    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9bee146-e705-478f-abda-fa3111705a69-istio-envoy\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv\" (UID: \"c9bee146-e705-478f-abda-fa3111705a69\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv"
Oct 30 05:13:51.812882 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:51.812852    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9bee146-e705-478f-abda-fa3111705a69-istio-podinfo\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv\" (UID: \"c9bee146-e705-478f-abda-fa3111705a69\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv"
Oct 30 05:13:51.828307 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:51.828280    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-659kd\" (UniqueName: \"kubernetes.io/projected/c9bee146-e705-478f-abda-fa3111705a69-kube-api-access-659kd\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv\" (UID: \"c9bee146-e705-478f-abda-fa3111705a69\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv"
Oct 30 05:13:51.828610 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:51.828588    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9bee146-e705-478f-abda-fa3111705a69-istio-token\") pod \"e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv\" (UID: \"c9bee146-e705-478f-abda-fa3111705a69\") " pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv"
Oct 30 05:13:51.958441 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:51.958346    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv"
Oct 30 05:13:52.282046 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:52.282016    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv"]
Oct 30 05:13:52.283373 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:13:52.283346    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9bee146_e705_478f_abda_fa3111705a69.slice/crio-135adfdd25dc44d93e850876fcf19e9938f7da98fdf994e1b1bf590adb1829ae WatchSource:0}: Error finding container 135adfdd25dc44d93e850876fcf19e9938f7da98fdf994e1b1bf590adb1829ae: Status 404 returned error can't find the container with id 135adfdd25dc44d93e850876fcf19e9938f7da98fdf994e1b1bf590adb1829ae
Oct 30 05:13:52.286248 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:52.286217    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:13:52.286313 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:52.286298    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:13:52.286351 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:52.286329    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:13:52.754981 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:52.754941    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv" event={"ID":"c9bee146-e705-478f-abda-fa3111705a69","Type":"ContainerStarted","Data":"035d0fbe0bbd1161ec30ac37a257da7946af6bbf1c76e8c83ffd7780ab336135"}
Oct 30 05:13:52.755153 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:52.755007    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv" event={"ID":"c9bee146-e705-478f-abda-fa3111705a69","Type":"ContainerStarted","Data":"135adfdd25dc44d93e850876fcf19e9938f7da98fdf994e1b1bf590adb1829ae"}
Oct 30 05:13:52.817382 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:52.817343    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:52.818536 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:52.818189    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv" podStartSLOduration=1.818169651 podStartE2EDuration="1.818169651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:13:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:13:52.805401284 +0000 UTC m=+1762.029221223" watchObservedRunningTime="2026-10-30 05:13:52.818169651 +0000 UTC m=+1762.041989552"
Oct 30 05:13:52.958967 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:52.958837    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv"
Oct 30 05:13:53.960563 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:53.960531    5898 patch_prober.go:29] interesting pod/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Oct 30 05:13:53.960983 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:53.960581    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv" podUID="c9bee146-e705-478f-abda-fa3111705a69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Oct 30 05:13:54.302187 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:54.302101    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ac96f3-7846ddb485-qfvcp"]
Oct 30 05:13:54.302626 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:54.302596    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac96f3-7846ddb485-qfvcp" podUID="a8c6428a-4956-430a-a835-8206dd6dcf9a" containerName="maas-api" containerID="cri-o://49458ce621e18223c3d30525ff077aa3b54d1d8ea8697748869d6ab71185fd72" gracePeriod=30
Oct 30 05:13:54.342923 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:54.342459    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:54.859198 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:54.856043    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac96f3-7846ddb485-qfvcp"
Oct 30 05:13:54.862180 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:54.859638    5898 generic.go:356] "Generic (PLEG): container finished" podID="a8c6428a-4956-430a-a835-8206dd6dcf9a" containerID="49458ce621e18223c3d30525ff077aa3b54d1d8ea8697748869d6ab71185fd72" exitCode=0
Oct 30 05:13:54.862180 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:54.860623    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac96f3-7846ddb485-qfvcp" event={"ID":"a8c6428a-4956-430a-a835-8206dd6dcf9a","Type":"ContainerDied","Data":"49458ce621e18223c3d30525ff077aa3b54d1d8ea8697748869d6ab71185fd72"}
Oct 30 05:13:54.862180 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:54.860681    5898 scope.go:122] "RemoveContainer" containerID="49458ce621e18223c3d30525ff077aa3b54d1d8ea8697748869d6ab71185fd72"
Oct 30 05:13:54.910922 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:54.906338    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl"]
Oct 30 05:13:54.910922 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:54.910144    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl" podUID="58083aef-e47e-4ba5-84f0-a70d80973181" containerName="istio-proxy" containerID="cri-o://c674cfaa68f9547135031751a24ff0d52cc962e6a7adf05a35e611623a887b76" gracePeriod=30
Oct 30 05:13:55.038490 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:55.022426    5898 patch_prober.go:29] interesting pod/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Oct 30 05:13:55.038490 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:55.027512    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv" podUID="c9bee146-e705-478f-abda-fa3111705a69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Oct 30 05:13:55.151928 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:55.140028    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:55.151928 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:55.140914    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8c6428a-4956-430a-a835-8206dd6dcf9a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8c6428a-4956-430a-a835-8206dd6dcf9a-metrics-certs\") pod \"a8c6428a-4956-430a-a835-8206dd6dcf9a\" (UID: \"a8c6428a-4956-430a-a835-8206dd6dcf9a\") "
Oct 30 05:13:55.151928 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:55.141008    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8c6428a-4956-430a-a835-8206dd6dcf9a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8c6428a-4956-430a-a835-8206dd6dcf9a-maas-api-tls\") pod \"a8c6428a-4956-430a-a835-8206dd6dcf9a\" (UID: \"a8c6428a-4956-430a-a835-8206dd6dcf9a\") "
Oct 30 05:13:55.151928 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:55.141079    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8c6428a-4956-430a-a835-8206dd6dcf9a-kube-api-access-4vxr8\" (UniqueName: \"kubernetes.io/projected/a8c6428a-4956-430a-a835-8206dd6dcf9a-kube-api-access-4vxr8\") pod \"a8c6428a-4956-430a-a835-8206dd6dcf9a\" (UID: \"a8c6428a-4956-430a-a835-8206dd6dcf9a\") "
Oct 30 05:13:55.151928 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:55.143795    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8c6428a-4956-430a-a835-8206dd6dcf9a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8c6428a-4956-430a-a835-8206dd6dcf9a-openshift-service-ca\") pod \"a8c6428a-4956-430a-a835-8206dd6dcf9a\" (UID: \"a8c6428a-4956-430a-a835-8206dd6dcf9a\") "
Oct 30 05:13:55.151928 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:55.144186    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8c6428a-4956-430a-a835-8206dd6dcf9a-openshift-service-ca" pod "a8c6428a-4956-430a-a835-8206dd6dcf9a" (UID: "a8c6428a-4956-430a-a835-8206dd6dcf9a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:13:55.151928 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:55.145330    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8c6428a-4956-430a-a835-8206dd6dcf9a-maas-api-tls" pod "a8c6428a-4956-430a-a835-8206dd6dcf9a" (UID: "a8c6428a-4956-430a-a835-8206dd6dcf9a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:13:55.166958 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:55.161228    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8c6428a-4956-430a-a835-8206dd6dcf9a-metrics-certs" pod "a8c6428a-4956-430a-a835-8206dd6dcf9a" (UID: "a8c6428a-4956-430a-a835-8206dd6dcf9a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:13:55.166958 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:55.161389    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8c6428a-4956-430a-a835-8206dd6dcf9a-kube-api-access-4vxr8" pod "a8c6428a-4956-430a-a835-8206dd6dcf9a" (UID: "a8c6428a-4956-430a-a835-8206dd6dcf9a"). InnerVolumeSpecName "kube-api-access-4vxr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:13:55.166958 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:55.165473    5898 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8c6428a-4956-430a-a835-8206dd6dcf9a-openshift-service-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:13:55.166958 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:55.165509    5898 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8c6428a-4956-430a-a835-8206dd6dcf9a-metrics-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:13:55.166958 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:55.165524    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8c6428a-4956-430a-a835-8206dd6dcf9a-maas-api-tls\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:13:55.166958 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:55.165536    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vxr8\" (UniqueName: \"kubernetes.io/projected/a8c6428a-4956-430a-a835-8206dd6dcf9a-kube-api-access-4vxr8\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:13:55.296429 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:55.296345    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:55.413210 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:55.412428    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:55.516172 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:55.516136    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:55.856846 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:55.856814    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:55.867869 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:55.867846    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac96f3-7846ddb485-qfvcp"
Oct 30 05:13:55.868022 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:55.867885    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac96f3-7846ddb485-qfvcp" event={"ID":"a8c6428a-4956-430a-a835-8206dd6dcf9a","Type":"ContainerDied","Data":"358752aac99eb1b9111761c964c33854cb8e7ec6433163db64fabfe2349e1b9c"}
Oct 30 05:13:55.932599 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:55.932560    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ac96f3-7846ddb485-qfvcp"]
Oct 30 05:13:55.939493 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:55.939421    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:55.943347 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:55.943325    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ac96f3-7846ddb485-qfvcp"]
Oct 30 05:13:55.959480 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:55.959458    5898 patch_prober.go:29] interesting pod/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Oct 30 05:13:55.959608 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:55.959507    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv" podUID="c9bee146-e705-478f-abda-fa3111705a69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Oct 30 05:13:56.842517 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:13:56.842477    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:13:56.910779 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:56.910739    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8c6428a-4956-430a-a835-8206dd6dcf9a" path="/var/lib/kubelet/pods/a8c6428a-4956-430a-a835-8206dd6dcf9a/volumes"
Oct 30 05:13:56.970334 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:56.965397    5898 patch_prober.go:29] interesting pod/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Oct 30 05:13:56.970334 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:56.965454    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv" podUID="c9bee146-e705-478f-abda-fa3111705a69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Oct 30 05:13:57.921065 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:57.921026    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:57.958711 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:57.958684    5898 patch_prober.go:29] interesting pod/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Oct 30 05:13:57.958863 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:57.958741    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv" podUID="c9bee146-e705-478f-abda-fa3111705a69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Oct 30 05:13:58.718188 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:58.718153    5898 patch_prober.go:29] interesting pod/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 05:13:58.718364 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:58.718206    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl" podUID="58083aef-e47e-4ba5-84f0-a70d80973181" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 05:13:58.998272 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:58.998178    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:59.099150 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:59.099117    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv"
Oct 30 05:13:59.100464 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:59.100437    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv"
Oct 30 05:13:59.101751 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:59.101726    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv"
Oct 30 05:13:59.103414 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:59.103392    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:13:59.482866 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:13:59.482833    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:00.065478 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:00.065446    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:00.194858 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:00.192986    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:00.351149 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:00.351121    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl"
Oct 30 05:14:00.418696 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:00.418664    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:00.455351 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:00.455322    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58083aef-e47e-4ba5-84f0-a70d80973181-kube-api-access-xtztk\" (UniqueName: \"kubernetes.io/projected/58083aef-e47e-4ba5-84f0-a70d80973181-kube-api-access-xtztk\") pod \"58083aef-e47e-4ba5-84f0-a70d80973181\" (UID: \"58083aef-e47e-4ba5-84f0-a70d80973181\") "
Oct 30 05:14:00.455530 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:00.455418    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58083aef-e47e-4ba5-84f0-a70d80973181-istio-token\" (UniqueName: \"kubernetes.io/projected/58083aef-e47e-4ba5-84f0-a70d80973181-istio-token\") pod \"58083aef-e47e-4ba5-84f0-a70d80973181\" (UID: \"58083aef-e47e-4ba5-84f0-a70d80973181\") "
Oct 30 05:14:00.455530 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:00.455438    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58083aef-e47e-4ba5-84f0-a70d80973181-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58083aef-e47e-4ba5-84f0-a70d80973181-istio-data\") pod \"58083aef-e47e-4ba5-84f0-a70d80973181\" (UID: \"58083aef-e47e-4ba5-84f0-a70d80973181\") "
Oct 30 05:14:00.455530 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:00.455512    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58083aef-e47e-4ba5-84f0-a70d80973181-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58083aef-e47e-4ba5-84f0-a70d80973181-credential-socket\") pod \"58083aef-e47e-4ba5-84f0-a70d80973181\" (UID: \"58083aef-e47e-4ba5-84f0-a70d80973181\") "
Oct 30 05:14:00.455717 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:00.455557    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58083aef-e47e-4ba5-84f0-a70d80973181-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58083aef-e47e-4ba5-84f0-a70d80973181-workload-socket\") pod \"58083aef-e47e-4ba5-84f0-a70d80973181\" (UID: \"58083aef-e47e-4ba5-84f0-a70d80973181\") "
Oct 30 05:14:00.455717 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:00.455598    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58083aef-e47e-4ba5-84f0-a70d80973181-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58083aef-e47e-4ba5-84f0-a70d80973181-istio-envoy\") pod \"58083aef-e47e-4ba5-84f0-a70d80973181\" (UID: \"58083aef-e47e-4ba5-84f0-a70d80973181\") "
Oct 30 05:14:00.455717 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:00.455617    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58083aef-e47e-4ba5-84f0-a70d80973181-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58083aef-e47e-4ba5-84f0-a70d80973181-istiod-ca-cert\") pod \"58083aef-e47e-4ba5-84f0-a70d80973181\" (UID: \"58083aef-e47e-4ba5-84f0-a70d80973181\") "
Oct 30 05:14:00.455717 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:00.455700    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58083aef-e47e-4ba5-84f0-a70d80973181-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58083aef-e47e-4ba5-84f0-a70d80973181-workload-certs\") pod \"58083aef-e47e-4ba5-84f0-a70d80973181\" (UID: \"58083aef-e47e-4ba5-84f0-a70d80973181\") "
Oct 30 05:14:00.455948 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:00.455727    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/58083aef-e47e-4ba5-84f0-a70d80973181-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58083aef-e47e-4ba5-84f0-a70d80973181-istio-podinfo\") pod \"58083aef-e47e-4ba5-84f0-a70d80973181\" (UID: \"58083aef-e47e-4ba5-84f0-a70d80973181\") "
Oct 30 05:14:00.455948 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:00.455836    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58083aef-e47e-4ba5-84f0-a70d80973181-istio-data" pod "58083aef-e47e-4ba5-84f0-a70d80973181" (UID: "58083aef-e47e-4ba5-84f0-a70d80973181"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:14:00.455948 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:00.455928    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58083aef-e47e-4ba5-84f0-a70d80973181-credential-socket" pod "58083aef-e47e-4ba5-84f0-a70d80973181" (UID: "58083aef-e47e-4ba5-84f0-a70d80973181"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:14:00.456118 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:00.455991    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58083aef-e47e-4ba5-84f0-a70d80973181-workload-socket" pod "58083aef-e47e-4ba5-84f0-a70d80973181" (UID: "58083aef-e47e-4ba5-84f0-a70d80973181"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:14:00.456118 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:00.456078    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58083aef-e47e-4ba5-84f0-a70d80973181-workload-certs" pod "58083aef-e47e-4ba5-84f0-a70d80973181" (UID: "58083aef-e47e-4ba5-84f0-a70d80973181"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:14:00.456348 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:00.456311    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58083aef-e47e-4ba5-84f0-a70d80973181-istiod-ca-cert" pod "58083aef-e47e-4ba5-84f0-a70d80973181" (UID: "58083aef-e47e-4ba5-84f0-a70d80973181"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:14:00.456528 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:00.456504    5898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58083aef-e47e-4ba5-84f0-a70d80973181-credential-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:14:00.456600 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:00.456533    5898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58083aef-e47e-4ba5-84f0-a70d80973181-workload-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:14:00.456600 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:00.456549    5898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58083aef-e47e-4ba5-84f0-a70d80973181-istiod-ca-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:14:00.456600 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:00.456564    5898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58083aef-e47e-4ba5-84f0-a70d80973181-workload-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:14:00.456600 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:00.456577    5898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58083aef-e47e-4ba5-84f0-a70d80973181-istio-data\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:14:00.457855 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:00.457828    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58083aef-e47e-4ba5-84f0-a70d80973181-istio-envoy" pod "58083aef-e47e-4ba5-84f0-a70d80973181" (UID: "58083aef-e47e-4ba5-84f0-a70d80973181"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:14:00.458015 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:00.457942    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58083aef-e47e-4ba5-84f0-a70d80973181-istio-token" pod "58083aef-e47e-4ba5-84f0-a70d80973181" (UID: "58083aef-e47e-4ba5-84f0-a70d80973181"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:14:00.458015 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:00.457978    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58083aef-e47e-4ba5-84f0-a70d80973181-kube-api-access-xtztk" pod "58083aef-e47e-4ba5-84f0-a70d80973181" (UID: "58083aef-e47e-4ba5-84f0-a70d80973181"). InnerVolumeSpecName "kube-api-access-xtztk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:14:00.458248 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:00.458219    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58083aef-e47e-4ba5-84f0-a70d80973181-istio-podinfo" pod "58083aef-e47e-4ba5-84f0-a70d80973181" (UID: "58083aef-e47e-4ba5-84f0-a70d80973181"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:14:00.519795 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:00.519764    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:00.557845 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:00.557814    5898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58083aef-e47e-4ba5-84f0-a70d80973181-istio-podinfo\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:14:00.557845 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:00.557841    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtztk\" (UniqueName: \"kubernetes.io/projected/58083aef-e47e-4ba5-84f0-a70d80973181-kube-api-access-xtztk\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:14:00.557845 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:00.557853    5898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58083aef-e47e-4ba5-84f0-a70d80973181-istio-token\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:14:00.558082 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:00.557862    5898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58083aef-e47e-4ba5-84f0-a70d80973181-istio-envoy\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:14:00.786594 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:00.786490    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:00.963452 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:00.946504    5898 generic.go:356] "Generic (PLEG): container finished" podID="58083aef-e47e-4ba5-84f0-a70d80973181" containerID="c674cfaa68f9547135031751a24ff0d52cc962e6a7adf05a35e611623a887b76" exitCode=0
Oct 30 05:14:00.963452 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:00.947706    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl"
Oct 30 05:14:00.963452 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:00.948369    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl" event={"ID":"58083aef-e47e-4ba5-84f0-a70d80973181","Type":"ContainerDied","Data":"c674cfaa68f9547135031751a24ff0d52cc962e6a7adf05a35e611623a887b76"}
Oct 30 05:14:00.963452 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:00.948416    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl" event={"ID":"58083aef-e47e-4ba5-84f0-a70d80973181","Type":"ContainerDied","Data":"d0cec78f9ef1eecb84315a0caf2fd27b673e9f1196d6e23e8c865e2ba8824124"}
Oct 30 05:14:00.963452 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:00.948443    5898 scope.go:122] "RemoveContainer" containerID="c674cfaa68f9547135031751a24ff0d52cc962e6a7adf05a35e611623a887b76"
Oct 30 05:14:00.963452 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:00.961877    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:01.033958 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:01.033652    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl"]
Oct 30 05:14:01.056975 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:01.055815    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-sgrgl"]
Oct 30 05:14:01.101216 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:01.101096    5898 scope.go:122] "RemoveContainer" containerID="c674cfaa68f9547135031751a24ff0d52cc962e6a7adf05a35e611623a887b76"
Oct 30 05:14:01.101762 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:14:01.101625    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c674cfaa68f9547135031751a24ff0d52cc962e6a7adf05a35e611623a887b76\": container with ID starting with c674cfaa68f9547135031751a24ff0d52cc962e6a7adf05a35e611623a887b76 not found: ID does not exist" containerID="c674cfaa68f9547135031751a24ff0d52cc962e6a7adf05a35e611623a887b76"
Oct 30 05:14:01.101762 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:01.101680    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c674cfaa68f9547135031751a24ff0d52cc962e6a7adf05a35e611623a887b76"} err="failed to get container status \"c674cfaa68f9547135031751a24ff0d52cc962e6a7adf05a35e611623a887b76\": rpc error: code = NotFound desc = could not find container \"c674cfaa68f9547135031751a24ff0d52cc962e6a7adf05a35e611623a887b76\": container with ID starting with c674cfaa68f9547135031751a24ff0d52cc962e6a7adf05a35e611623a887b76 not found: ID does not exist"
Oct 30 05:14:02.118417 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:02.118381    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:02.251139 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:02.251098    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:02.362089 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:02.362058    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:02.892916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:02.892865    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58083aef-e47e-4ba5-84f0-a70d80973181" path="/var/lib/kubelet/pods/58083aef-e47e-4ba5-84f0-a70d80973181/volumes"
Oct 30 05:14:03.202421 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:03.202325    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:03.452948 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:03.452823    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:03.558964 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:03.558929    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:03.671716 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:03.671679    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:03.811671 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:03.811588    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5"]
Oct 30 05:14:03.812086 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:03.812056    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5" podUID="0ec75360-8dfb-45e6-85cc-267e154966f8" containerName="istio-proxy" containerID="cri-o://ea7341b122eebb48a4b4b17e748b6c0a9b798e32d75efae9cee6cc6e91be2f64" gracePeriod=30
Oct 30 05:14:03.837806 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:03.837774    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:04.112269 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:04.112232    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:04.211125 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:04.211064    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:04.423404 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:04.423315    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:04.550061 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:04.550013    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:04.671239 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:04.671207    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:04.771871 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:04.771789    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:05.470812 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:05.470758    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:06.402026 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:06.401994    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:06.501076 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:06.501042    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:06.816544 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:06.816451    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:07.850656 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:07.850618    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:08.223058 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:08.222972    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:08.329807 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:08.329773    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:08.897879 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:08.897849    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:09.012980 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:09.012921    5898 generic.go:356] "Generic (PLEG): container finished" podID="0ec75360-8dfb-45e6-85cc-267e154966f8" containerID="ea7341b122eebb48a4b4b17e748b6c0a9b798e32d75efae9cee6cc6e91be2f64" exitCode=0
Oct 30 05:14:09.013155 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:09.012993    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5" event={"ID":"0ec75360-8dfb-45e6-85cc-267e154966f8","Type":"ContainerDied","Data":"ea7341b122eebb48a4b4b17e748b6c0a9b798e32d75efae9cee6cc6e91be2f64"}
Oct 30 05:14:09.013155 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:09.013065    5898 scope.go:122] "RemoveContainer" containerID="dde1623074ec63842dffea36914a2b3fc9704c56c5df4478d23b263b1b8c50f8"
Oct 30 05:14:09.119531 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:09.119504    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5"
Oct 30 05:14:09.197728 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:09.197691    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ec75360-8dfb-45e6-85cc-267e154966f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ec75360-8dfb-45e6-85cc-267e154966f8-istiod-ca-cert\") pod \"0ec75360-8dfb-45e6-85cc-267e154966f8\" (UID: \"0ec75360-8dfb-45e6-85cc-267e154966f8\") "
Oct 30 05:14:09.197930 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:09.197779    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ec75360-8dfb-45e6-85cc-267e154966f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ec75360-8dfb-45e6-85cc-267e154966f8-workload-certs\") pod \"0ec75360-8dfb-45e6-85cc-267e154966f8\" (UID: \"0ec75360-8dfb-45e6-85cc-267e154966f8\") "
Oct 30 05:14:09.197930 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:09.197856    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ec75360-8dfb-45e6-85cc-267e154966f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ec75360-8dfb-45e6-85cc-267e154966f8-credential-socket\") pod \"0ec75360-8dfb-45e6-85cc-267e154966f8\" (UID: \"0ec75360-8dfb-45e6-85cc-267e154966f8\") "
Oct 30 05:14:09.198061 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:09.197974    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ec75360-8dfb-45e6-85cc-267e154966f8-istio-token\" (UniqueName: \"kubernetes.io/projected/0ec75360-8dfb-45e6-85cc-267e154966f8-istio-token\") pod \"0ec75360-8dfb-45e6-85cc-267e154966f8\" (UID: \"0ec75360-8dfb-45e6-85cc-267e154966f8\") "
Oct 30 05:14:09.198172 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:09.198141    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ec75360-8dfb-45e6-85cc-267e154966f8-istiod-ca-cert" pod "0ec75360-8dfb-45e6-85cc-267e154966f8" (UID: "0ec75360-8dfb-45e6-85cc-267e154966f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:14:09.198259 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:09.198199    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ec75360-8dfb-45e6-85cc-267e154966f8-workload-certs" pod "0ec75360-8dfb-45e6-85cc-267e154966f8" (UID: "0ec75360-8dfb-45e6-85cc-267e154966f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:14:09.198328 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:09.198266    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ec75360-8dfb-45e6-85cc-267e154966f8-credential-socket" pod "0ec75360-8dfb-45e6-85cc-267e154966f8" (UID: "0ec75360-8dfb-45e6-85cc-267e154966f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:14:09.198411 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:09.198006    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ec75360-8dfb-45e6-85cc-267e154966f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ec75360-8dfb-45e6-85cc-267e154966f8-istio-envoy\") pod \"0ec75360-8dfb-45e6-85cc-267e154966f8\" (UID: \"0ec75360-8dfb-45e6-85cc-267e154966f8\") "
Oct 30 05:14:09.198485 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:09.198464    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ec75360-8dfb-45e6-85cc-267e154966f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ec75360-8dfb-45e6-85cc-267e154966f8-istio-podinfo\") pod \"0ec75360-8dfb-45e6-85cc-267e154966f8\" (UID: \"0ec75360-8dfb-45e6-85cc-267e154966f8\") "
Oct 30 05:14:09.198560 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:09.198532    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ec75360-8dfb-45e6-85cc-267e154966f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ec75360-8dfb-45e6-85cc-267e154966f8-istio-data\") pod \"0ec75360-8dfb-45e6-85cc-267e154966f8\" (UID: \"0ec75360-8dfb-45e6-85cc-267e154966f8\") "
Oct 30 05:14:09.198635 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:09.198618    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ec75360-8dfb-45e6-85cc-267e154966f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ec75360-8dfb-45e6-85cc-267e154966f8-workload-socket\") pod \"0ec75360-8dfb-45e6-85cc-267e154966f8\" (UID: \"0ec75360-8dfb-45e6-85cc-267e154966f8\") "
Oct 30 05:14:09.198703 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:09.198670    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ec75360-8dfb-45e6-85cc-267e154966f8-kube-api-access-z42ch\" (UniqueName: \"kubernetes.io/projected/0ec75360-8dfb-45e6-85cc-267e154966f8-kube-api-access-z42ch\") pod \"0ec75360-8dfb-45e6-85cc-267e154966f8\" (UID: \"0ec75360-8dfb-45e6-85cc-267e154966f8\") "
Oct 30 05:14:09.199192 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:09.199162    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ec75360-8dfb-45e6-85cc-267e154966f8-workload-socket" pod "0ec75360-8dfb-45e6-85cc-267e154966f8" (UID: "0ec75360-8dfb-45e6-85cc-267e154966f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:14:09.199321 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:09.199298    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ec75360-8dfb-45e6-85cc-267e154966f8-istio-data" pod "0ec75360-8dfb-45e6-85cc-267e154966f8" (UID: "0ec75360-8dfb-45e6-85cc-267e154966f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:14:09.199824 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:09.199800    5898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ec75360-8dfb-45e6-85cc-267e154966f8-workload-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:14:09.199961 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:09.199829    5898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ec75360-8dfb-45e6-85cc-267e154966f8-istiod-ca-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:14:09.199961 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:09.199843    5898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ec75360-8dfb-45e6-85cc-267e154966f8-workload-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:14:09.199961 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:09.199855    5898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ec75360-8dfb-45e6-85cc-267e154966f8-credential-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:14:09.199961 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:09.199868    5898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ec75360-8dfb-45e6-85cc-267e154966f8-istio-data\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:14:09.201070 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:09.201041    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ec75360-8dfb-45e6-85cc-267e154966f8-istio-envoy" pod "0ec75360-8dfb-45e6-85cc-267e154966f8" (UID: "0ec75360-8dfb-45e6-85cc-267e154966f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:14:09.201198 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:09.201179    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ec75360-8dfb-45e6-85cc-267e154966f8-istio-token" pod "0ec75360-8dfb-45e6-85cc-267e154966f8" (UID: "0ec75360-8dfb-45e6-85cc-267e154966f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:14:09.201687 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:09.201662    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ec75360-8dfb-45e6-85cc-267e154966f8-istio-podinfo" pod "0ec75360-8dfb-45e6-85cc-267e154966f8" (UID: "0ec75360-8dfb-45e6-85cc-267e154966f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:14:09.201857 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:09.201834    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ec75360-8dfb-45e6-85cc-267e154966f8-kube-api-access-z42ch" pod "0ec75360-8dfb-45e6-85cc-267e154966f8" (UID: "0ec75360-8dfb-45e6-85cc-267e154966f8"). InnerVolumeSpecName "kube-api-access-z42ch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:14:09.301754 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:09.301722    5898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ec75360-8dfb-45e6-85cc-267e154966f8-istio-token\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:14:09.301754 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:09.301748    5898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ec75360-8dfb-45e6-85cc-267e154966f8-istio-envoy\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:14:09.301754 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:09.301757    5898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ec75360-8dfb-45e6-85cc-267e154966f8-istio-podinfo\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:14:09.302048 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:09.301766    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z42ch\" (UniqueName: \"kubernetes.io/projected/0ec75360-8dfb-45e6-85cc-267e154966f8-kube-api-access-z42ch\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:14:09.967310 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:09.967275    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:10.021421 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:10.021387    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5" event={"ID":"0ec75360-8dfb-45e6-85cc-267e154966f8","Type":"ContainerDied","Data":"75a637b30f6e1b11c80765f374f4e1c0c7dc1406dd50a1323497387c77bee2db"}
Oct 30 05:14:10.021421 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:10.021408    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5"
Oct 30 05:14:10.021656 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:10.021451    5898 scope.go:122] "RemoveContainer" containerID="ea7341b122eebb48a4b4b17e748b6c0a9b798e32d75efae9cee6cc6e91be2f64"
Oct 30 05:14:10.065703 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:10.065675    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:10.096554 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:10.096515    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5"]
Oct 30 05:14:10.107267 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:10.107241    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-ac96f3-openshift-default-86c6f75967-b8tz5"]
Oct 30 05:14:10.744753 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:10.744717    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:10.891462 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:10.891430    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ec75360-8dfb-45e6-85cc-267e154966f8" path="/var/lib/kubelet/pods/0ec75360-8dfb-45e6-85cc-267e154966f8/volumes"
Oct 30 05:14:11.457040 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:11.457006    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:11.621004 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:11.620969    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:11.722110 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:11.722018    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:12.157465 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:12.157435    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:13.627678 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:13.627643    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:15.226583 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:15.226545    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4225df-6bd7847ff6-bdtpc"]
Oct 30 05:14:15.348424 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:15.348393    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:15.433065 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:15.433034    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:15.515719 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:15.515590    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:15.722244 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:15.722215    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:17.444669 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:17.444631    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:19.214917 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:19.214874    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:19.346117 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:19.346083    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:19.445818 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:19.445785    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:19.699527 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:19.699482    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:20.227529 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:20.227487    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-4225df-6bd7847ff6-bdtpc" podUID="63f37c81-e93d-4e27-ad10-9665c4951666" containerName="payload-processing" containerID="cri-o://0a45b2f93b89eb67ac1bc4eb74b1ca4b2ac879501cdd832e4ce3aeb814130e54" gracePeriod=25
Oct 30 05:14:20.495817 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:20.495745    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4225df-6bd7847ff6-bdtpc"
Oct 30 05:14:20.542245 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:20.542212    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:20.597216 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:20.597181    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63f37c81-e93d-4e27-ad10-9665c4951666-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63f37c81-e93d-4e27-ad10-9665c4951666-plugins-config-volume\") pod \"63f37c81-e93d-4e27-ad10-9665c4951666\" (UID: \"63f37c81-e93d-4e27-ad10-9665c4951666\") "
Oct 30 05:14:20.597497 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:20.597477    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63f37c81-e93d-4e27-ad10-9665c4951666-kube-api-access-qzbw2\" (UniqueName: \"kubernetes.io/projected/63f37c81-e93d-4e27-ad10-9665c4951666-kube-api-access-qzbw2\") pod \"63f37c81-e93d-4e27-ad10-9665c4951666\" (UID: \"63f37c81-e93d-4e27-ad10-9665c4951666\") "
Oct 30 05:14:20.597585 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:20.597560    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63f37c81-e93d-4e27-ad10-9665c4951666-plugins-config-volume" pod "63f37c81-e93d-4e27-ad10-9665c4951666" (UID: "63f37c81-e93d-4e27-ad10-9665c4951666"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:14:20.598017 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:20.597998    5898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63f37c81-e93d-4e27-ad10-9665c4951666-plugins-config-volume\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:14:20.599544 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:20.599522    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63f37c81-e93d-4e27-ad10-9665c4951666-kube-api-access-qzbw2" pod "63f37c81-e93d-4e27-ad10-9665c4951666" (UID: "63f37c81-e93d-4e27-ad10-9665c4951666"). InnerVolumeSpecName "kube-api-access-qzbw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:14:20.699634 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:20.699607    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzbw2\" (UniqueName: \"kubernetes.io/projected/63f37c81-e93d-4e27-ad10-9665c4951666-kube-api-access-qzbw2\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:14:20.972472 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:20.972412    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:21.000649 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:21.000615    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w"]
Oct 30 05:14:21.001291 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:21.001270    5898 cpu_manager.go:414] "containerMap: removing container" podUID="0ec75360-8dfb-45e6-85cc-267e154966f8" containerName="istio-proxy"
Oct 30 05:14:21.001291 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:21.001290    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ec75360-8dfb-45e6-85cc-267e154966f8" containerName="istio-proxy"
Oct 30 05:14:21.001476 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:21.001313    5898 cpu_manager.go:414] "containerMap: removing container" podUID="0ec75360-8dfb-45e6-85cc-267e154966f8" containerName="istio-proxy"
Oct 30 05:14:21.001476 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:21.001318    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ec75360-8dfb-45e6-85cc-267e154966f8" containerName="istio-proxy"
Oct 30 05:14:21.001476 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:21.001327    5898 cpu_manager.go:414] "containerMap: removing container" podUID="58083aef-e47e-4ba5-84f0-a70d80973181" containerName="istio-proxy"
Oct 30 05:14:21.001476 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:21.001332    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58083aef-e47e-4ba5-84f0-a70d80973181" containerName="istio-proxy"
Oct 30 05:14:21.001476 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:21.001353    5898 cpu_manager.go:414] "containerMap: removing container" podUID="63f37c81-e93d-4e27-ad10-9665c4951666" containerName="payload-processing"
Oct 30 05:14:21.001476 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:21.001358    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63f37c81-e93d-4e27-ad10-9665c4951666" containerName="payload-processing"
Oct 30 05:14:21.001476 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:21.001367    5898 cpu_manager.go:414] "containerMap: removing container" podUID="a8c6428a-4956-430a-a835-8206dd6dcf9a" containerName="maas-api"
Oct 30 05:14:21.001476 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:21.001372    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8c6428a-4956-430a-a835-8206dd6dcf9a" containerName="maas-api"
Oct 30 05:14:21.001476 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:21.001472    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ec75360-8dfb-45e6-85cc-267e154966f8" containerName="istio-proxy"
Oct 30 05:14:21.001476 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:21.001480    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="58083aef-e47e-4ba5-84f0-a70d80973181" containerName="istio-proxy"
Oct 30 05:14:21.002033 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:21.001490    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="63f37c81-e93d-4e27-ad10-9665c4951666" containerName="payload-processing"
Oct 30 05:14:21.002033 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:21.001498    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8c6428a-4956-430a-a835-8206dd6dcf9a" containerName="maas-api"
Oct 30 05:14:21.005280 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:21.005261    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w"
Oct 30 05:14:21.029009 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:21.028853    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w"]
Oct 30 05:14:21.033221 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:21.033189    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-356cae-openshift-default-dockercfg-ss9zj\""
Oct 30 05:14:21.065761 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:21.065728    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:21.108411 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:21.108375    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a999b0b-fe75-49de-af7f-bdc3145966c7-istio-data\") pod \"e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w\" (UID: \"8a999b0b-fe75-49de-af7f-bdc3145966c7\") " pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w"
Oct 30 05:14:21.108616 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:21.108454    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzwtw\" (UniqueName: \"kubernetes.io/projected/8a999b0b-fe75-49de-af7f-bdc3145966c7-kube-api-access-lzwtw\") pod \"e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w\" (UID: \"8a999b0b-fe75-49de-af7f-bdc3145966c7\") " pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w"
Oct 30 05:14:21.108616 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:21.108501    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a999b0b-fe75-49de-af7f-bdc3145966c7-credential-socket\") pod \"e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w\" (UID: \"8a999b0b-fe75-49de-af7f-bdc3145966c7\") " pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w"
Oct 30 05:14:21.108616 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:21.108536    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a999b0b-fe75-49de-af7f-bdc3145966c7-istiod-ca-cert\") pod \"e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w\" (UID: \"8a999b0b-fe75-49de-af7f-bdc3145966c7\") " pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w"
Oct 30 05:14:21.108616 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:21.108584    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a999b0b-fe75-49de-af7f-bdc3145966c7-workload-certs\") pod \"e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w\" (UID: \"8a999b0b-fe75-49de-af7f-bdc3145966c7\") " pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w"
Oct 30 05:14:21.108845 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:21.108652    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a999b0b-fe75-49de-af7f-bdc3145966c7-istio-envoy\") pod \"e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w\" (UID: \"8a999b0b-fe75-49de-af7f-bdc3145966c7\") " pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w"
Oct 30 05:14:21.108845 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:21.108676    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a999b0b-fe75-49de-af7f-bdc3145966c7-istio-token\") pod \"e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w\" (UID: \"8a999b0b-fe75-49de-af7f-bdc3145966c7\") " pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w"
Oct 30 05:14:21.108845 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:21.108732    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a999b0b-fe75-49de-af7f-bdc3145966c7-istio-podinfo\") pod \"e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w\" (UID: \"8a999b0b-fe75-49de-af7f-bdc3145966c7\") " pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w"
Oct 30 05:14:21.108845 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:21.108801    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a999b0b-fe75-49de-af7f-bdc3145966c7-workload-socket\") pod \"e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w\" (UID: \"8a999b0b-fe75-49de-af7f-bdc3145966c7\") " pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w"
Oct 30 05:14:21.118065 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:21.118011    5898 generic.go:356] "Generic (PLEG): container finished" podID="63f37c81-e93d-4e27-ad10-9665c4951666" containerID="0a45b2f93b89eb67ac1bc4eb74b1ca4b2ac879501cdd832e4ce3aeb814130e54" exitCode=0
Oct 30 05:14:21.118230 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:21.118076    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4225df-6bd7847ff6-bdtpc"
Oct 30 05:14:21.118646 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:21.118068    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4225df-6bd7847ff6-bdtpc" event={"ID":"63f37c81-e93d-4e27-ad10-9665c4951666","Type":"ContainerDied","Data":"0a45b2f93b89eb67ac1bc4eb74b1ca4b2ac879501cdd832e4ce3aeb814130e54"}
Oct 30 05:14:21.118809 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:21.118681    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4225df-6bd7847ff6-bdtpc" event={"ID":"63f37c81-e93d-4e27-ad10-9665c4951666","Type":"ContainerDied","Data":"2fa0d0aa59d4ebaee6fbe92d77d0af3bb3b6e8cc431093c2f03cd94fce1ab2ac"}
Oct 30 05:14:21.118809 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:21.118716    5898 scope.go:122] "RemoveContainer" containerID="0a45b2f93b89eb67ac1bc4eb74b1ca4b2ac879501cdd832e4ce3aeb814130e54"
Oct 30 05:14:21.213691 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:21.210174    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a999b0b-fe75-49de-af7f-bdc3145966c7-istio-data\") pod \"e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w\" (UID: \"8a999b0b-fe75-49de-af7f-bdc3145966c7\") " pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w"
Oct 30 05:14:21.213691 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:21.210270    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzwtw\" (UniqueName: \"kubernetes.io/projected/8a999b0b-fe75-49de-af7f-bdc3145966c7-kube-api-access-lzwtw\") pod \"e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w\" (UID: \"8a999b0b-fe75-49de-af7f-bdc3145966c7\") " pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w"
Oct 30 05:14:21.213691 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:21.210324    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a999b0b-fe75-49de-af7f-bdc3145966c7-credential-socket\") pod \"e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w\" (UID: \"8a999b0b-fe75-49de-af7f-bdc3145966c7\") " pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w"
Oct 30 05:14:21.213691 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:21.210359    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a999b0b-fe75-49de-af7f-bdc3145966c7-istiod-ca-cert\") pod \"e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w\" (UID: \"8a999b0b-fe75-49de-af7f-bdc3145966c7\") " pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w"
Oct 30 05:14:21.213691 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:21.210411    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a999b0b-fe75-49de-af7f-bdc3145966c7-workload-certs\") pod \"e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w\" (UID: \"8a999b0b-fe75-49de-af7f-bdc3145966c7\") " pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w"
Oct 30 05:14:21.213691 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:21.210480    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a999b0b-fe75-49de-af7f-bdc3145966c7-istio-envoy\") pod \"e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w\" (UID: \"8a999b0b-fe75-49de-af7f-bdc3145966c7\") " pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w"
Oct 30 05:14:21.213691 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:21.210503    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a999b0b-fe75-49de-af7f-bdc3145966c7-istio-token\") pod \"e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w\" (UID: \"8a999b0b-fe75-49de-af7f-bdc3145966c7\") " pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w"
Oct 30 05:14:21.213691 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:21.210557    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a999b0b-fe75-49de-af7f-bdc3145966c7-istio-podinfo\") pod \"e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w\" (UID: \"8a999b0b-fe75-49de-af7f-bdc3145966c7\") " pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w"
Oct 30 05:14:21.213691 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:21.210622    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a999b0b-fe75-49de-af7f-bdc3145966c7-workload-socket\") pod \"e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w\" (UID: \"8a999b0b-fe75-49de-af7f-bdc3145966c7\") " pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w"
Oct 30 05:14:21.213691 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:21.211180    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a999b0b-fe75-49de-af7f-bdc3145966c7-workload-socket\") pod \"e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w\" (UID: \"8a999b0b-fe75-49de-af7f-bdc3145966c7\") " pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w"
Oct 30 05:14:21.213691 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:21.211415    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a999b0b-fe75-49de-af7f-bdc3145966c7-istio-data\") pod \"e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w\" (UID: \"8a999b0b-fe75-49de-af7f-bdc3145966c7\") " pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w"
Oct 30 05:14:21.213691 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:21.211958    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a999b0b-fe75-49de-af7f-bdc3145966c7-credential-socket\") pod \"e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w\" (UID: \"8a999b0b-fe75-49de-af7f-bdc3145966c7\") " pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w"
Oct 30 05:14:21.213691 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:21.212559    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a999b0b-fe75-49de-af7f-bdc3145966c7-istiod-ca-cert\") pod \"e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w\" (UID: \"8a999b0b-fe75-49de-af7f-bdc3145966c7\") " pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w"
Oct 30 05:14:21.213691 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:21.212792    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a999b0b-fe75-49de-af7f-bdc3145966c7-workload-certs\") pod \"e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w\" (UID: \"8a999b0b-fe75-49de-af7f-bdc3145966c7\") " pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w"
Oct 30 05:14:21.227862 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:21.220848    5898 scope.go:122] "RemoveContainer" containerID="0a45b2f93b89eb67ac1bc4eb74b1ca4b2ac879501cdd832e4ce3aeb814130e54"
Oct 30 05:14:21.227862 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:14:21.221422    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a45b2f93b89eb67ac1bc4eb74b1ca4b2ac879501cdd832e4ce3aeb814130e54\": container with ID starting with 0a45b2f93b89eb67ac1bc4eb74b1ca4b2ac879501cdd832e4ce3aeb814130e54 not found: ID does not exist" containerID="0a45b2f93b89eb67ac1bc4eb74b1ca4b2ac879501cdd832e4ce3aeb814130e54"
Oct 30 05:14:21.227862 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:21.221455    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a45b2f93b89eb67ac1bc4eb74b1ca4b2ac879501cdd832e4ce3aeb814130e54"} err="failed to get container status \"0a45b2f93b89eb67ac1bc4eb74b1ca4b2ac879501cdd832e4ce3aeb814130e54\": rpc error: code = NotFound desc = could not find container \"0a45b2f93b89eb67ac1bc4eb74b1ca4b2ac879501cdd832e4ce3aeb814130e54\": container with ID starting with 0a45b2f93b89eb67ac1bc4eb74b1ca4b2ac879501cdd832e4ce3aeb814130e54 not found: ID does not exist"
Oct 30 05:14:21.240564 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:21.240480    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a999b0b-fe75-49de-af7f-bdc3145966c7-istio-podinfo\") pod \"e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w\" (UID: \"8a999b0b-fe75-49de-af7f-bdc3145966c7\") " pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w"
Oct 30 05:14:21.269739 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:21.253456    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4225df-6bd7847ff6-bdtpc"]
Oct 30 05:14:21.269739 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:21.268428    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a999b0b-fe75-49de-af7f-bdc3145966c7-istio-envoy\") pod \"e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w\" (UID: \"8a999b0b-fe75-49de-af7f-bdc3145966c7\") " pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w"
Oct 30 05:14:21.269739 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:21.268916    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a999b0b-fe75-49de-af7f-bdc3145966c7-istio-token\") pod \"e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w\" (UID: \"8a999b0b-fe75-49de-af7f-bdc3145966c7\") " pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w"
Oct 30 05:14:21.290677 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:21.290600    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzwtw\" (UniqueName: \"kubernetes.io/projected/8a999b0b-fe75-49de-af7f-bdc3145966c7-kube-api-access-lzwtw\") pod \"e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w\" (UID: \"8a999b0b-fe75-49de-af7f-bdc3145966c7\") " pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w"
Oct 30 05:14:21.299876 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:21.299839    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4225df-6bd7847ff6-bdtpc"]
Oct 30 05:14:21.331444 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:21.331155    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w"
Oct 30 05:14:21.541066 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:21.540918    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:21.692406 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:21.692369    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w"]
Oct 30 05:14:21.693956 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:14:21.693927    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a999b0b_fe75_49de_af7f_bdc3145966c7.slice/crio-495031a0f11906c630cb64d5f4a829aee63dbb2bed624b486eb2a1742e8ecab0 WatchSource:0}: Error finding container 495031a0f11906c630cb64d5f4a829aee63dbb2bed624b486eb2a1742e8ecab0: Status 404 returned error can't find the container with id 495031a0f11906c630cb64d5f4a829aee63dbb2bed624b486eb2a1742e8ecab0
Oct 30 05:14:21.697403 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:21.697371    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:14:21.697476 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:21.697451    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:14:21.697514 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:21.697482    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:14:22.128912 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:22.128852    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w" event={"ID":"8a999b0b-fe75-49de-af7f-bdc3145966c7","Type":"ContainerStarted","Data":"59e7172c0e4cddcdc33c748a06b25fba80835c7d5e4cd741ef64bc5dfe9b03e5"}
Oct 30 05:14:22.129119 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:22.128939    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w" event={"ID":"8a999b0b-fe75-49de-af7f-bdc3145966c7","Type":"ContainerStarted","Data":"495031a0f11906c630cb64d5f4a829aee63dbb2bed624b486eb2a1742e8ecab0"}
Oct 30 05:14:22.193036 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:22.192986    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w" podStartSLOduration=2.192972754 podStartE2EDuration="2.192972754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:14:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:14:22.179711966 +0000 UTC m=+1791.403531891" watchObservedRunningTime="2026-10-30 05:14:22.192972754 +0000 UTC m=+1791.416792656"
Oct 30 05:14:22.333922 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:22.333253    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w"
Oct 30 05:14:22.342416 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:22.342387    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w"
Oct 30 05:14:22.848103 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:22.848066    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:22.894709 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:22.894672    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63f37c81-e93d-4e27-ad10-9665c4951666" path="/var/lib/kubelet/pods/63f37c81-e93d-4e27-ad10-9665c4951666/volumes"
Oct 30 05:14:23.136775 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:23.136698    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w"
Oct 30 05:14:23.137607 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:23.137586    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w"
Oct 30 05:14:24.692384 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:24.692350    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:24.796981 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:24.796951    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:24.897764 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:24.897727    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:25.205096 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:25.205064    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:26.406766 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:26.406734    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:26.504990 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:26.504958    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:26.870548 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:14:26.870514    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:14:26.913073 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:26.913041    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:28.867311 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:28.867277    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:29.348923 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:29.348043    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:14:30.584913 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:30.584879    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:32.316072 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:32.316036    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:34.035378 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:34.035339    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:35.641300 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:35.641267    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:37.228424 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:37.228387    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:37.902554 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:37.902521    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:38.111712 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:38.111677    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:38.255441 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:38.255356    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:38.662953 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:38.662895    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:40.260727 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:40.260694    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7"]
Oct 30 05:14:40.261311 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:40.260949    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7" podUID="4e9e9b05-1135-4cde-8a29-42c7291befe3" containerName="istio-proxy" containerID="cri-o://f8752ce9d7ac11ac1013196c5b9ae5e8ab29ed555c946ffa0469a0382b95e89b" gracePeriod=30
Oct 30 05:14:40.268957 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:40.268887    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:40.272657 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:40.272630    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q"]
Oct 30 05:14:40.272912 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:40.272861    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q" podUID="3e97be36-ccbe-43d0-8566-dfcca1d8f6a0" containerName="istio-proxy" containerID="cri-o://55a438e6b58925a6b419c43e1ca1c5832e3469205bf4c56454f28a1d30a84dad" gracePeriod=30
Oct 30 05:14:40.962909 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:40.962863    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:42.083419 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:42.083347    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:43.496731 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:43.496684    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:44.008753 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:44.008711    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:44.843028 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:44.842996    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-c7f962/test-model-c7f962-kserve-5fd5595fd9-jbf8l"]
Oct 30 05:14:44.843776 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:44.843756    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ec75360-8dfb-45e6-85cc-267e154966f8" containerName="istio-proxy"
Oct 30 05:14:44.854423 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:44.854397    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c7f962/test-model-c7f962-kserve-5fd5595fd9-jbf8l"
Oct 30 05:14:44.868919 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:44.868401    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c7f962/test-model-c7f962-kserve-5fd5595fd9-jbf8l"]
Oct 30 05:14:44.873088 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:44.873062    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c7f962\"/\"openshift-service-ca.crt\""
Oct 30 05:14:44.873233 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:44.873153    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c7f962\"/\"kube-root-ca.crt\""
Oct 30 05:14:44.873289 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:44.873240    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c7f962\"/\"default-dockercfg-wbpbg\""
Oct 30 05:14:44.879697 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:44.879674    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c7f962\"/\"test-model-c7f962-kserve-self-signed-certs\""
Oct 30 05:14:44.881376 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:44.881352    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:44.916939 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:44.916872    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9aed90e-03b0-49b0-b0c5-509fd95836d6-tmp-dir\") pod \"test-model-c7f962-kserve-5fd5595fd9-jbf8l\" (UID: \"e9aed90e-03b0-49b0-b0c5-509fd95836d6\") " pod="ai-tenant-e2e-inf-a-c7f962/test-model-c7f962-kserve-5fd5595fd9-jbf8l"
Oct 30 05:14:44.917083 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:44.917044    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9aed90e-03b0-49b0-b0c5-509fd95836d6-tls-certs\") pod \"test-model-c7f962-kserve-5fd5595fd9-jbf8l\" (UID: \"e9aed90e-03b0-49b0-b0c5-509fd95836d6\") " pod="ai-tenant-e2e-inf-a-c7f962/test-model-c7f962-kserve-5fd5595fd9-jbf8l"
Oct 30 05:14:44.917150 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:44.917131    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwpxx\" (UniqueName: \"kubernetes.io/projected/e9aed90e-03b0-49b0-b0c5-509fd95836d6-kube-api-access-nwpxx\") pod \"test-model-c7f962-kserve-5fd5595fd9-jbf8l\" (UID: \"e9aed90e-03b0-49b0-b0c5-509fd95836d6\") " pod="ai-tenant-e2e-inf-a-c7f962/test-model-c7f962-kserve-5fd5595fd9-jbf8l"
Oct 30 05:14:44.917316 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:44.917294    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9aed90e-03b0-49b0-b0c5-509fd95836d6-dshm\") pod \"test-model-c7f962-kserve-5fd5595fd9-jbf8l\" (UID: \"e9aed90e-03b0-49b0-b0c5-509fd95836d6\") " pod="ai-tenant-e2e-inf-a-c7f962/test-model-c7f962-kserve-5fd5595fd9-jbf8l"
Oct 30 05:14:44.917389 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:44.917374    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9aed90e-03b0-49b0-b0c5-509fd95836d6-home\") pod \"test-model-c7f962-kserve-5fd5595fd9-jbf8l\" (UID: \"e9aed90e-03b0-49b0-b0c5-509fd95836d6\") " pod="ai-tenant-e2e-inf-a-c7f962/test-model-c7f962-kserve-5fd5595fd9-jbf8l"
Oct 30 05:14:44.917440 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:44.917407    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9aed90e-03b0-49b0-b0c5-509fd95836d6-model-cache\") pod \"test-model-c7f962-kserve-5fd5595fd9-jbf8l\" (UID: \"e9aed90e-03b0-49b0-b0c5-509fd95836d6\") " pod="ai-tenant-e2e-inf-a-c7f962/test-model-c7f962-kserve-5fd5595fd9-jbf8l"
Oct 30 05:14:45.019438 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.019404    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9aed90e-03b0-49b0-b0c5-509fd95836d6-tmp-dir\") pod \"test-model-c7f962-kserve-5fd5595fd9-jbf8l\" (UID: \"e9aed90e-03b0-49b0-b0c5-509fd95836d6\") " pod="ai-tenant-e2e-inf-a-c7f962/test-model-c7f962-kserve-5fd5595fd9-jbf8l"
Oct 30 05:14:45.019667 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.019492    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9aed90e-03b0-49b0-b0c5-509fd95836d6-tls-certs\") pod \"test-model-c7f962-kserve-5fd5595fd9-jbf8l\" (UID: \"e9aed90e-03b0-49b0-b0c5-509fd95836d6\") " pod="ai-tenant-e2e-inf-a-c7f962/test-model-c7f962-kserve-5fd5595fd9-jbf8l"
Oct 30 05:14:45.019667 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.019560    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwpxx\" (UniqueName: \"kubernetes.io/projected/e9aed90e-03b0-49b0-b0c5-509fd95836d6-kube-api-access-nwpxx\") pod \"test-model-c7f962-kserve-5fd5595fd9-jbf8l\" (UID: \"e9aed90e-03b0-49b0-b0c5-509fd95836d6\") " pod="ai-tenant-e2e-inf-a-c7f962/test-model-c7f962-kserve-5fd5595fd9-jbf8l"
Oct 30 05:14:45.019790 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.019698    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9aed90e-03b0-49b0-b0c5-509fd95836d6-dshm\") pod \"test-model-c7f962-kserve-5fd5595fd9-jbf8l\" (UID: \"e9aed90e-03b0-49b0-b0c5-509fd95836d6\") " pod="ai-tenant-e2e-inf-a-c7f962/test-model-c7f962-kserve-5fd5595fd9-jbf8l"
Oct 30 05:14:45.019790 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.019769    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9aed90e-03b0-49b0-b0c5-509fd95836d6-home\") pod \"test-model-c7f962-kserve-5fd5595fd9-jbf8l\" (UID: \"e9aed90e-03b0-49b0-b0c5-509fd95836d6\") " pod="ai-tenant-e2e-inf-a-c7f962/test-model-c7f962-kserve-5fd5595fd9-jbf8l"
Oct 30 05:14:45.019892 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.019801    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9aed90e-03b0-49b0-b0c5-509fd95836d6-model-cache\") pod \"test-model-c7f962-kserve-5fd5595fd9-jbf8l\" (UID: \"e9aed90e-03b0-49b0-b0c5-509fd95836d6\") " pod="ai-tenant-e2e-inf-a-c7f962/test-model-c7f962-kserve-5fd5595fd9-jbf8l"
Oct 30 05:14:45.020289 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.020264    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9aed90e-03b0-49b0-b0c5-509fd95836d6-model-cache\") pod \"test-model-c7f962-kserve-5fd5595fd9-jbf8l\" (UID: \"e9aed90e-03b0-49b0-b0c5-509fd95836d6\") " pod="ai-tenant-e2e-inf-a-c7f962/test-model-c7f962-kserve-5fd5595fd9-jbf8l"
Oct 30 05:14:45.020520 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.020502    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9aed90e-03b0-49b0-b0c5-509fd95836d6-tmp-dir\") pod \"test-model-c7f962-kserve-5fd5595fd9-jbf8l\" (UID: \"e9aed90e-03b0-49b0-b0c5-509fd95836d6\") " pod="ai-tenant-e2e-inf-a-c7f962/test-model-c7f962-kserve-5fd5595fd9-jbf8l"
Oct 30 05:14:45.020860 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.020829    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9aed90e-03b0-49b0-b0c5-509fd95836d6-home\") pod \"test-model-c7f962-kserve-5fd5595fd9-jbf8l\" (UID: \"e9aed90e-03b0-49b0-b0c5-509fd95836d6\") " pod="ai-tenant-e2e-inf-a-c7f962/test-model-c7f962-kserve-5fd5595fd9-jbf8l"
Oct 30 05:14:45.022998 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.022981    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9aed90e-03b0-49b0-b0c5-509fd95836d6-dshm\") pod \"test-model-c7f962-kserve-5fd5595fd9-jbf8l\" (UID: \"e9aed90e-03b0-49b0-b0c5-509fd95836d6\") " pod="ai-tenant-e2e-inf-a-c7f962/test-model-c7f962-kserve-5fd5595fd9-jbf8l"
Oct 30 05:14:45.023402 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.023380    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9aed90e-03b0-49b0-b0c5-509fd95836d6-tls-certs\") pod \"test-model-c7f962-kserve-5fd5595fd9-jbf8l\" (UID: \"e9aed90e-03b0-49b0-b0c5-509fd95836d6\") " pod="ai-tenant-e2e-inf-a-c7f962/test-model-c7f962-kserve-5fd5595fd9-jbf8l"
Oct 30 05:14:45.039003 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.038981    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwpxx\" (UniqueName: \"kubernetes.io/projected/e9aed90e-03b0-49b0-b0c5-509fd95836d6-kube-api-access-nwpxx\") pod \"test-model-c7f962-kserve-5fd5595fd9-jbf8l\" (UID: \"e9aed90e-03b0-49b0-b0c5-509fd95836d6\") " pod="ai-tenant-e2e-inf-a-c7f962/test-model-c7f962-kserve-5fd5595fd9-jbf8l"
Oct 30 05:14:45.174989 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.174884    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c7f962/test-model-c7f962-kserve-5fd5595fd9-jbf8l"
Oct 30 05:14:45.336764 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.335725    5898 patch_prober.go:29] interesting pod/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 05:14:45.336764 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.335797    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7" podUID="4e9e9b05-1135-4cde-8a29-42c7291befe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 05:14:45.457961 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.457852    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:45.557755 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.557725    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:45.584680 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.584654    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7"
Oct 30 05:14:45.584933 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:14:45.584882    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode9aed90e_03b0_49b0_b0c5_509fd95836d6.slice/crio-21efe5157a54b0079e21b8cbdb971de03f11da3afe49bb67aa50ff51eb751912 WatchSource:0}: Error finding container 21efe5157a54b0079e21b8cbdb971de03f11da3afe49bb67aa50ff51eb751912: Status 404 returned error can't find the container with id 21efe5157a54b0079e21b8cbdb971de03f11da3afe49bb67aa50ff51eb751912
Oct 30 05:14:45.604712 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.604681    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c7f962/test-model-c7f962-kserve-5fd5595fd9-jbf8l"]
Oct 30 05:14:45.634965 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.634488    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e9e9b05-1135-4cde-8a29-42c7291befe3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e9e9b05-1135-4cde-8a29-42c7291befe3-istiod-ca-cert\") pod \"4e9e9b05-1135-4cde-8a29-42c7291befe3\" (UID: \"4e9e9b05-1135-4cde-8a29-42c7291befe3\") "
Oct 30 05:14:45.634965 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.634678    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e9e9b05-1135-4cde-8a29-42c7291befe3-istio-token\" (UniqueName: \"kubernetes.io/projected/4e9e9b05-1135-4cde-8a29-42c7291befe3-istio-token\") pod \"4e9e9b05-1135-4cde-8a29-42c7291befe3\" (UID: \"4e9e9b05-1135-4cde-8a29-42c7291befe3\") "
Oct 30 05:14:45.634965 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.634768    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e9e9b05-1135-4cde-8a29-42c7291befe3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e9e9b05-1135-4cde-8a29-42c7291befe3-workload-certs\") pod \"4e9e9b05-1135-4cde-8a29-42c7291befe3\" (UID: \"4e9e9b05-1135-4cde-8a29-42c7291befe3\") "
Oct 30 05:14:45.634965 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.634813    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e9e9b05-1135-4cde-8a29-42c7291befe3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e9e9b05-1135-4cde-8a29-42c7291befe3-istio-data\") pod \"4e9e9b05-1135-4cde-8a29-42c7291befe3\" (UID: \"4e9e9b05-1135-4cde-8a29-42c7291befe3\") "
Oct 30 05:14:45.634965 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.634876    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e9e9b05-1135-4cde-8a29-42c7291befe3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e9e9b05-1135-4cde-8a29-42c7291befe3-workload-socket\") pod \"4e9e9b05-1135-4cde-8a29-42c7291befe3\" (UID: \"4e9e9b05-1135-4cde-8a29-42c7291befe3\") "
Oct 30 05:14:45.636587 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.635002    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e9e9b05-1135-4cde-8a29-42c7291befe3-kube-api-access-z42bm\" (UniqueName: \"kubernetes.io/projected/4e9e9b05-1135-4cde-8a29-42c7291befe3-kube-api-access-z42bm\") pod \"4e9e9b05-1135-4cde-8a29-42c7291befe3\" (UID: \"4e9e9b05-1135-4cde-8a29-42c7291befe3\") "
Oct 30 05:14:45.636587 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.635073    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e9e9b05-1135-4cde-8a29-42c7291befe3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e9e9b05-1135-4cde-8a29-42c7291befe3-istio-podinfo\") pod \"4e9e9b05-1135-4cde-8a29-42c7291befe3\" (UID: \"4e9e9b05-1135-4cde-8a29-42c7291befe3\") "
Oct 30 05:14:45.636587 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.635237    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e9e9b05-1135-4cde-8a29-42c7291befe3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e9e9b05-1135-4cde-8a29-42c7291befe3-istio-envoy\") pod \"4e9e9b05-1135-4cde-8a29-42c7291befe3\" (UID: \"4e9e9b05-1135-4cde-8a29-42c7291befe3\") "
Oct 30 05:14:45.636587 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.635267    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e9e9b05-1135-4cde-8a29-42c7291befe3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e9e9b05-1135-4cde-8a29-42c7291befe3-credential-socket\") pod \"4e9e9b05-1135-4cde-8a29-42c7291befe3\" (UID: \"4e9e9b05-1135-4cde-8a29-42c7291befe3\") "
Oct 30 05:14:45.636587 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.634841    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e9e9b05-1135-4cde-8a29-42c7291befe3-istiod-ca-cert" pod "4e9e9b05-1135-4cde-8a29-42c7291befe3" (UID: "4e9e9b05-1135-4cde-8a29-42c7291befe3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:14:45.636587 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.635468    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e9e9b05-1135-4cde-8a29-42c7291befe3-workload-socket" pod "4e9e9b05-1135-4cde-8a29-42c7291befe3" (UID: "4e9e9b05-1135-4cde-8a29-42c7291befe3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:14:45.636587 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.635963    5898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e9e9b05-1135-4cde-8a29-42c7291befe3-istiod-ca-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:14:45.636587 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.635983    5898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e9e9b05-1135-4cde-8a29-42c7291befe3-workload-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:14:45.637022 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.636776    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e9e9b05-1135-4cde-8a29-42c7291befe3-istio-data" pod "4e9e9b05-1135-4cde-8a29-42c7291befe3" (UID: "4e9e9b05-1135-4cde-8a29-42c7291befe3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:14:45.637139 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.637113    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e9e9b05-1135-4cde-8a29-42c7291befe3-workload-certs" pod "4e9e9b05-1135-4cde-8a29-42c7291befe3" (UID: "4e9e9b05-1135-4cde-8a29-42c7291befe3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:14:45.641821 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.641794    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e9e9b05-1135-4cde-8a29-42c7291befe3-credential-socket" pod "4e9e9b05-1135-4cde-8a29-42c7291befe3" (UID: "4e9e9b05-1135-4cde-8a29-42c7291befe3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:14:45.641943 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.641922    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e9e9b05-1135-4cde-8a29-42c7291befe3-istio-envoy" pod "4e9e9b05-1135-4cde-8a29-42c7291befe3" (UID: "4e9e9b05-1135-4cde-8a29-42c7291befe3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:14:45.642826 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.642801    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e9e9b05-1135-4cde-8a29-42c7291befe3-istio-token" pod "4e9e9b05-1135-4cde-8a29-42c7291befe3" (UID: "4e9e9b05-1135-4cde-8a29-42c7291befe3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:14:45.645671 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.645632    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e9e9b05-1135-4cde-8a29-42c7291befe3-istio-podinfo" pod "4e9e9b05-1135-4cde-8a29-42c7291befe3" (UID: "4e9e9b05-1135-4cde-8a29-42c7291befe3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:14:45.646960 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.646053    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q"
Oct 30 05:14:45.647242 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.647207    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e9e9b05-1135-4cde-8a29-42c7291befe3-kube-api-access-z42bm" pod "4e9e9b05-1135-4cde-8a29-42c7291befe3" (UID: "4e9e9b05-1135-4cde-8a29-42c7291befe3"). InnerVolumeSpecName "kube-api-access-z42bm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:14:45.737510 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.736652    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e97be36-ccbe-43d0-8566-dfcca1d8f6a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e97be36-ccbe-43d0-8566-dfcca1d8f6a0-istiod-ca-cert\") pod \"3e97be36-ccbe-43d0-8566-dfcca1d8f6a0\" (UID: \"3e97be36-ccbe-43d0-8566-dfcca1d8f6a0\") "
Oct 30 05:14:45.737510 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.736770    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e97be36-ccbe-43d0-8566-dfcca1d8f6a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e97be36-ccbe-43d0-8566-dfcca1d8f6a0-istio-podinfo\") pod \"3e97be36-ccbe-43d0-8566-dfcca1d8f6a0\" (UID: \"3e97be36-ccbe-43d0-8566-dfcca1d8f6a0\") "
Oct 30 05:14:45.737510 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.736815    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e97be36-ccbe-43d0-8566-dfcca1d8f6a0-istio-token\" (UniqueName: \"kubernetes.io/projected/3e97be36-ccbe-43d0-8566-dfcca1d8f6a0-istio-token\") pod \"3e97be36-ccbe-43d0-8566-dfcca1d8f6a0\" (UID: \"3e97be36-ccbe-43d0-8566-dfcca1d8f6a0\") "
Oct 30 05:14:45.737510 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.736870    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e97be36-ccbe-43d0-8566-dfcca1d8f6a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e97be36-ccbe-43d0-8566-dfcca1d8f6a0-istio-data\") pod \"3e97be36-ccbe-43d0-8566-dfcca1d8f6a0\" (UID: \"3e97be36-ccbe-43d0-8566-dfcca1d8f6a0\") "
Oct 30 05:14:45.737510 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.736930    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e97be36-ccbe-43d0-8566-dfcca1d8f6a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e97be36-ccbe-43d0-8566-dfcca1d8f6a0-istio-envoy\") pod \"3e97be36-ccbe-43d0-8566-dfcca1d8f6a0\" (UID: \"3e97be36-ccbe-43d0-8566-dfcca1d8f6a0\") "
Oct 30 05:14:45.737510 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.737035    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e97be36-ccbe-43d0-8566-dfcca1d8f6a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e97be36-ccbe-43d0-8566-dfcca1d8f6a0-workload-socket\") pod \"3e97be36-ccbe-43d0-8566-dfcca1d8f6a0\" (UID: \"3e97be36-ccbe-43d0-8566-dfcca1d8f6a0\") "
Oct 30 05:14:45.738828 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.738647    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e97be36-ccbe-43d0-8566-dfcca1d8f6a0-workload-socket" pod "3e97be36-ccbe-43d0-8566-dfcca1d8f6a0" (UID: "3e97be36-ccbe-43d0-8566-dfcca1d8f6a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:14:45.739239 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.739019    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e97be36-ccbe-43d0-8566-dfcca1d8f6a0-istio-data" pod "3e97be36-ccbe-43d0-8566-dfcca1d8f6a0" (UID: "3e97be36-ccbe-43d0-8566-dfcca1d8f6a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:14:45.739539 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.739517    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e97be36-ccbe-43d0-8566-dfcca1d8f6a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e97be36-ccbe-43d0-8566-dfcca1d8f6a0-credential-socket\") pod \"3e97be36-ccbe-43d0-8566-dfcca1d8f6a0\" (UID: \"3e97be36-ccbe-43d0-8566-dfcca1d8f6a0\") "
Oct 30 05:14:45.739709 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.739691    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e97be36-ccbe-43d0-8566-dfcca1d8f6a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e97be36-ccbe-43d0-8566-dfcca1d8f6a0-workload-certs\") pod \"3e97be36-ccbe-43d0-8566-dfcca1d8f6a0\" (UID: \"3e97be36-ccbe-43d0-8566-dfcca1d8f6a0\") "
Oct 30 05:14:45.739855 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.739838    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e97be36-ccbe-43d0-8566-dfcca1d8f6a0-kube-api-access-t2qpm\" (UniqueName: \"kubernetes.io/projected/3e97be36-ccbe-43d0-8566-dfcca1d8f6a0-kube-api-access-t2qpm\") pod \"3e97be36-ccbe-43d0-8566-dfcca1d8f6a0\" (UID: \"3e97be36-ccbe-43d0-8566-dfcca1d8f6a0\") "
Oct 30 05:14:45.740534 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.740131    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e97be36-ccbe-43d0-8566-dfcca1d8f6a0-istiod-ca-cert" pod "3e97be36-ccbe-43d0-8566-dfcca1d8f6a0" (UID: "3e97be36-ccbe-43d0-8566-dfcca1d8f6a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:14:45.740708 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.740679    5898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e97be36-ccbe-43d0-8566-dfcca1d8f6a0-istio-data\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:14:45.740825 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.740716    5898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e9e9b05-1135-4cde-8a29-42c7291befe3-istio-token\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:14:45.740825 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.740731    5898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e97be36-ccbe-43d0-8566-dfcca1d8f6a0-workload-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:14:45.740825 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.740745    5898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e9e9b05-1135-4cde-8a29-42c7291befe3-workload-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:14:45.740825 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.740761    5898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e9e9b05-1135-4cde-8a29-42c7291befe3-istio-data\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:14:45.740825 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.740772    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z42bm\" (UniqueName: \"kubernetes.io/projected/4e9e9b05-1135-4cde-8a29-42c7291befe3-kube-api-access-z42bm\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:14:45.740825 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.740786    5898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e9e9b05-1135-4cde-8a29-42c7291befe3-istio-podinfo\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:14:45.740825 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.740800    5898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e9e9b05-1135-4cde-8a29-42c7291befe3-istio-envoy\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:14:45.740825 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.740813    5898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e9e9b05-1135-4cde-8a29-42c7291befe3-credential-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:14:45.740825 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.740825    5898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e97be36-ccbe-43d0-8566-dfcca1d8f6a0-istiod-ca-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:14:45.744037 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.742964    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e97be36-ccbe-43d0-8566-dfcca1d8f6a0-credential-socket" pod "3e97be36-ccbe-43d0-8566-dfcca1d8f6a0" (UID: "3e97be36-ccbe-43d0-8566-dfcca1d8f6a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:14:45.744037 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.743190    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e97be36-ccbe-43d0-8566-dfcca1d8f6a0-workload-certs" pod "3e97be36-ccbe-43d0-8566-dfcca1d8f6a0" (UID: "3e97be36-ccbe-43d0-8566-dfcca1d8f6a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:14:45.744037 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.743255    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e97be36-ccbe-43d0-8566-dfcca1d8f6a0-istio-podinfo" pod "3e97be36-ccbe-43d0-8566-dfcca1d8f6a0" (UID: "3e97be36-ccbe-43d0-8566-dfcca1d8f6a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:14:45.744037 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.743542    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e97be36-ccbe-43d0-8566-dfcca1d8f6a0-kube-api-access-t2qpm" pod "3e97be36-ccbe-43d0-8566-dfcca1d8f6a0" (UID: "3e97be36-ccbe-43d0-8566-dfcca1d8f6a0"). InnerVolumeSpecName "kube-api-access-t2qpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:14:45.746430 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.746401    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e97be36-ccbe-43d0-8566-dfcca1d8f6a0-istio-token" pod "3e97be36-ccbe-43d0-8566-dfcca1d8f6a0" (UID: "3e97be36-ccbe-43d0-8566-dfcca1d8f6a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:14:45.748167 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.748142    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e97be36-ccbe-43d0-8566-dfcca1d8f6a0-istio-envoy" pod "3e97be36-ccbe-43d0-8566-dfcca1d8f6a0" (UID: "3e97be36-ccbe-43d0-8566-dfcca1d8f6a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:14:45.831962 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.831895    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:45.841858 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.841833    5898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e97be36-ccbe-43d0-8566-dfcca1d8f6a0-credential-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:14:45.841858 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.841857    5898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e97be36-ccbe-43d0-8566-dfcca1d8f6a0-workload-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:14:45.842085 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.841868    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2qpm\" (UniqueName: \"kubernetes.io/projected/3e97be36-ccbe-43d0-8566-dfcca1d8f6a0-kube-api-access-t2qpm\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:14:45.842085 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.841878    5898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e97be36-ccbe-43d0-8566-dfcca1d8f6a0-istio-podinfo\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:14:45.842085 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.841886    5898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e97be36-ccbe-43d0-8566-dfcca1d8f6a0-istio-token\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:14:45.842085 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.841894    5898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e97be36-ccbe-43d0-8566-dfcca1d8f6a0-istio-envoy\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:14:45.931341 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:45.931290    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:46.034394 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:46.034273    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:46.312120 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:46.312036    5898 generic.go:356] "Generic (PLEG): container finished" podID="3e97be36-ccbe-43d0-8566-dfcca1d8f6a0" containerID="55a438e6b58925a6b419c43e1ca1c5832e3469205bf4c56454f28a1d30a84dad" exitCode=0
Oct 30 05:14:46.312120 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:46.312090    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q"
Oct 30 05:14:46.312341 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:46.312087    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q" event={"ID":"3e97be36-ccbe-43d0-8566-dfcca1d8f6a0","Type":"ContainerDied","Data":"55a438e6b58925a6b419c43e1ca1c5832e3469205bf4c56454f28a1d30a84dad"}
Oct 30 05:14:46.312341 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:46.312254    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q" event={"ID":"3e97be36-ccbe-43d0-8566-dfcca1d8f6a0","Type":"ContainerDied","Data":"f1bc6127affc41189b98e4ad6dc32c70b9917546f04c49ccacd1fc402776e071"}
Oct 30 05:14:46.312341 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:46.312280    5898 scope.go:122] "RemoveContainer" containerID="55a438e6b58925a6b419c43e1ca1c5832e3469205bf4c56454f28a1d30a84dad"
Oct 30 05:14:46.313984 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:46.313956    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c7f962/test-model-c7f962-kserve-5fd5595fd9-jbf8l" event={"ID":"e9aed90e-03b0-49b0-b0c5-509fd95836d6","Type":"ContainerStarted","Data":"4c2780b744eb506d5817cb75fc0902320f6b7537315eeda90c07dc5efef54fb0"}
Oct 30 05:14:46.314100 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:46.314001    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c7f962/test-model-c7f962-kserve-5fd5595fd9-jbf8l" event={"ID":"e9aed90e-03b0-49b0-b0c5-509fd95836d6","Type":"ContainerStarted","Data":"21efe5157a54b0079e21b8cbdb971de03f11da3afe49bb67aa50ff51eb751912"}
Oct 30 05:14:46.315485 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:46.315462    5898 generic.go:356] "Generic (PLEG): container finished" podID="4e9e9b05-1135-4cde-8a29-42c7291befe3" containerID="f8752ce9d7ac11ac1013196c5b9ae5e8ab29ed555c946ffa0469a0382b95e89b" exitCode=0
Oct 30 05:14:46.315613 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:46.315504    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7"
Oct 30 05:14:46.315613 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:46.315525    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7" event={"ID":"4e9e9b05-1135-4cde-8a29-42c7291befe3","Type":"ContainerDied","Data":"f8752ce9d7ac11ac1013196c5b9ae5e8ab29ed555c946ffa0469a0382b95e89b"}
Oct 30 05:14:46.315613 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:46.315553    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7" event={"ID":"4e9e9b05-1135-4cde-8a29-42c7291befe3","Type":"ContainerDied","Data":"a0ace625ab33db4bfc95255dcc7d9e69b3b7d541c966b1a9076c23110dd592b2"}
Oct 30 05:14:46.325041 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:46.325019    5898 scope.go:122] "RemoveContainer" containerID="55a438e6b58925a6b419c43e1ca1c5832e3469205bf4c56454f28a1d30a84dad"
Oct 30 05:14:46.325408 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:14:46.325351    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55a438e6b58925a6b419c43e1ca1c5832e3469205bf4c56454f28a1d30a84dad\": container with ID starting with 55a438e6b58925a6b419c43e1ca1c5832e3469205bf4c56454f28a1d30a84dad not found: ID does not exist" containerID="55a438e6b58925a6b419c43e1ca1c5832e3469205bf4c56454f28a1d30a84dad"
Oct 30 05:14:46.325408 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:46.325384    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55a438e6b58925a6b419c43e1ca1c5832e3469205bf4c56454f28a1d30a84dad"} err="failed to get container status \"55a438e6b58925a6b419c43e1ca1c5832e3469205bf4c56454f28a1d30a84dad\": rpc error: code = NotFound desc = could not find container \"55a438e6b58925a6b419c43e1ca1c5832e3469205bf4c56454f28a1d30a84dad\": container with ID starting with 55a438e6b58925a6b419c43e1ca1c5832e3469205bf4c56454f28a1d30a84dad not found: ID does not exist"
Oct 30 05:14:46.326098 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:46.325415    5898 scope.go:122] "RemoveContainer" containerID="f8752ce9d7ac11ac1013196c5b9ae5e8ab29ed555c946ffa0469a0382b95e89b"
Oct 30 05:14:46.340104 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:46.340087    5898 scope.go:122] "RemoveContainer" containerID="f8752ce9d7ac11ac1013196c5b9ae5e8ab29ed555c946ffa0469a0382b95e89b"
Oct 30 05:14:46.340351 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:14:46.340332    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8752ce9d7ac11ac1013196c5b9ae5e8ab29ed555c946ffa0469a0382b95e89b\": container with ID starting with f8752ce9d7ac11ac1013196c5b9ae5e8ab29ed555c946ffa0469a0382b95e89b not found: ID does not exist" containerID="f8752ce9d7ac11ac1013196c5b9ae5e8ab29ed555c946ffa0469a0382b95e89b"
Oct 30 05:14:46.340402 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:46.340358    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8752ce9d7ac11ac1013196c5b9ae5e8ab29ed555c946ffa0469a0382b95e89b"} err="failed to get container status \"f8752ce9d7ac11ac1013196c5b9ae5e8ab29ed555c946ffa0469a0382b95e89b\": rpc error: code = NotFound desc = could not find container \"f8752ce9d7ac11ac1013196c5b9ae5e8ab29ed555c946ffa0469a0382b95e89b\": container with ID starting with f8752ce9d7ac11ac1013196c5b9ae5e8ab29ed555c946ffa0469a0382b95e89b not found: ID does not exist"
Oct 30 05:14:46.365124 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:46.365081    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-c7f962/test-model-c7f962-kserve-5fd5595fd9-jbf8l" podStartSLOduration=2.365065545 podStartE2EDuration="2.365065545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:14:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:14:46.362996161 +0000 UTC m=+1815.586816103" watchObservedRunningTime="2026-10-30 05:14:46.365065545 +0000 UTC m=+1815.588885448"
Oct 30 05:14:46.413912 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:46.413877    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q"]
Oct 30 05:14:46.424637 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:46.424614    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-rf49q"]
Oct 30 05:14:46.449834 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:46.449802    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7"]
Oct 30 05:14:46.454396 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:46.454369    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-pqmc7"]
Oct 30 05:14:46.893401 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:46.893360    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e97be36-ccbe-43d0-8566-dfcca1d8f6a0" path="/var/lib/kubelet/pods/3e97be36-ccbe-43d0-8566-dfcca1d8f6a0/volumes"
Oct 30 05:14:46.894118 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:46.894096    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e9e9b05-1135-4cde-8a29-42c7291befe3" path="/var/lib/kubelet/pods/4e9e9b05-1135-4cde-8a29-42c7291befe3/volumes"
Oct 30 05:14:47.177116 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:47.177034    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-c7f962/test-model-c7f962-kserve-5fd5595fd9-jbf8l"
Oct 30 05:14:47.897195 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:47.897160    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:49.808201 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:49.808170    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:51.440479 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:51.440438    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8afb26-688849cfdf-6b6c5"]
Oct 30 05:14:51.441328 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:51.441310    5898 cpu_manager.go:414] "containerMap: removing container" podUID="4e9e9b05-1135-4cde-8a29-42c7291befe3" containerName="istio-proxy"
Oct 30 05:14:51.441386 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:51.441331    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e9e9b05-1135-4cde-8a29-42c7291befe3" containerName="istio-proxy"
Oct 30 05:14:51.441424 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:51.441400    5898 cpu_manager.go:414] "containerMap: removing container" podUID="3e97be36-ccbe-43d0-8566-dfcca1d8f6a0" containerName="istio-proxy"
Oct 30 05:14:51.441424 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:51.441409    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e97be36-ccbe-43d0-8566-dfcca1d8f6a0" containerName="istio-proxy"
Oct 30 05:14:51.441540 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:51.441528    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e97be36-ccbe-43d0-8566-dfcca1d8f6a0" containerName="istio-proxy"
Oct 30 05:14:51.441578 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:51.441548    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e9e9b05-1135-4cde-8a29-42c7291befe3" containerName="istio-proxy"
Oct 30 05:14:51.447032 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:51.447010    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8afb26-688849cfdf-6b6c5"
Oct 30 05:14:51.463764 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:51.463738    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8afb26-688849cfdf-6b6c5"]
Oct 30 05:14:51.471000 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:51.470970    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-8afb26-serving-cert\""
Oct 30 05:14:51.512829 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:51.512784    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:51.535722 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:51.535692    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpp82\" (UniqueName: \"kubernetes.io/projected/96952584-fdf3-4731-a845-83ec28cd69b4-kube-api-access-qpp82\") pod \"maas-api-e2e-inf-b-8afb26-688849cfdf-6b6c5\" (UID: \"96952584-fdf3-4731-a845-83ec28cd69b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8afb26-688849cfdf-6b6c5"
Oct 30 05:14:51.535939 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:51.535883    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96952584-fdf3-4731-a845-83ec28cd69b4-maas-api-tls\") pod \"maas-api-e2e-inf-b-8afb26-688849cfdf-6b6c5\" (UID: \"96952584-fdf3-4731-a845-83ec28cd69b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8afb26-688849cfdf-6b6c5"
Oct 30 05:14:51.536073 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:51.536047    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96952584-fdf3-4731-a845-83ec28cd69b4-metrics-certs\") pod \"maas-api-e2e-inf-b-8afb26-688849cfdf-6b6c5\" (UID: \"96952584-fdf3-4731-a845-83ec28cd69b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8afb26-688849cfdf-6b6c5"
Oct 30 05:14:51.536167 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:51.536104    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96952584-fdf3-4731-a845-83ec28cd69b4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8afb26-688849cfdf-6b6c5\" (UID: \"96952584-fdf3-4731-a845-83ec28cd69b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8afb26-688849cfdf-6b6c5"
Oct 30 05:14:51.613741 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:51.613712    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:51.651504 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:51.651459    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96952584-fdf3-4731-a845-83ec28cd69b4-metrics-certs\") pod \"maas-api-e2e-inf-b-8afb26-688849cfdf-6b6c5\" (UID: \"96952584-fdf3-4731-a845-83ec28cd69b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8afb26-688849cfdf-6b6c5"
Oct 30 05:14:51.651753 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:51.651531    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96952584-fdf3-4731-a845-83ec28cd69b4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8afb26-688849cfdf-6b6c5\" (UID: \"96952584-fdf3-4731-a845-83ec28cd69b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8afb26-688849cfdf-6b6c5"
Oct 30 05:14:51.651753 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:51.651713    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpp82\" (UniqueName: \"kubernetes.io/projected/96952584-fdf3-4731-a845-83ec28cd69b4-kube-api-access-qpp82\") pod \"maas-api-e2e-inf-b-8afb26-688849cfdf-6b6c5\" (UID: \"96952584-fdf3-4731-a845-83ec28cd69b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8afb26-688849cfdf-6b6c5"
Oct 30 05:14:51.651893 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:51.651849    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96952584-fdf3-4731-a845-83ec28cd69b4-maas-api-tls\") pod \"maas-api-e2e-inf-b-8afb26-688849cfdf-6b6c5\" (UID: \"96952584-fdf3-4731-a845-83ec28cd69b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8afb26-688849cfdf-6b6c5"
Oct 30 05:14:51.652108 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:14:51.652081    5898 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-8afb26-serving-cert: secret "maas-api-e2e-inf-b-8afb26-serving-cert" not found
Oct 30 05:14:51.652197 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:14:51.652160    5898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96952584-fdf3-4731-a845-83ec28cd69b4-maas-api-tls podName:96952584-fdf3-4731-a845-83ec28cd69b4 nodeName:}" failed. No retries permitted until 2026-10-30 05:14:52.1521389 +0000 UTC m=+1821.375958783 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/96952584-fdf3-4731-a845-83ec28cd69b4-maas-api-tls") pod "maas-api-e2e-inf-b-8afb26-688849cfdf-6b6c5" (UID: "96952584-fdf3-4731-a845-83ec28cd69b4") : secret "maas-api-e2e-inf-b-8afb26-serving-cert" not found
Oct 30 05:14:51.652959 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:51.652930    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96952584-fdf3-4731-a845-83ec28cd69b4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8afb26-688849cfdf-6b6c5\" (UID: \"96952584-fdf3-4731-a845-83ec28cd69b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8afb26-688849cfdf-6b6c5"
Oct 30 05:14:51.655064 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:51.655044    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96952584-fdf3-4731-a845-83ec28cd69b4-metrics-certs\") pod \"maas-api-e2e-inf-b-8afb26-688849cfdf-6b6c5\" (UID: \"96952584-fdf3-4731-a845-83ec28cd69b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8afb26-688849cfdf-6b6c5"
Oct 30 05:14:51.674999 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:51.674967    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpp82\" (UniqueName: \"kubernetes.io/projected/96952584-fdf3-4731-a845-83ec28cd69b4-kube-api-access-qpp82\") pod \"maas-api-e2e-inf-b-8afb26-688849cfdf-6b6c5\" (UID: \"96952584-fdf3-4731-a845-83ec28cd69b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8afb26-688849cfdf-6b6c5"
Oct 30 05:14:51.890842 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:51.890805    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:52.158767 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:52.158695    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96952584-fdf3-4731-a845-83ec28cd69b4-maas-api-tls\") pod \"maas-api-e2e-inf-b-8afb26-688849cfdf-6b6c5\" (UID: \"96952584-fdf3-4731-a845-83ec28cd69b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8afb26-688849cfdf-6b6c5"
Oct 30 05:14:52.161208 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:52.161181    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96952584-fdf3-4731-a845-83ec28cd69b4-maas-api-tls\") pod \"maas-api-e2e-inf-b-8afb26-688849cfdf-6b6c5\" (UID: \"96952584-fdf3-4731-a845-83ec28cd69b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8afb26-688849cfdf-6b6c5"
Oct 30 05:14:52.361273 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:52.361239    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8afb26-688849cfdf-6b6c5"
Oct 30 05:14:52.675156 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:52.675120    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8afb26-688849cfdf-6b6c5"]
Oct 30 05:14:52.677005 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:14:52.676940    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96952584_fdf3_4731_a845_83ec28cd69b4.slice/crio-f889aa3073e97f7404cf8ffa328de86fb5dd9f3fd441631300228c0930b36c91 WatchSource:0}: Error finding container f889aa3073e97f7404cf8ffa328de86fb5dd9f3fd441631300228c0930b36c91: Status 404 returned error can't find the container with id f889aa3073e97f7404cf8ffa328de86fb5dd9f3fd441631300228c0930b36c91
Oct 30 05:14:52.678723 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:52.678700    5898 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 05:14:53.322717 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:53.322692    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:53.388095 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:53.388046    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8afb26-688849cfdf-6b6c5" event={"ID":"96952584-fdf3-4731-a845-83ec28cd69b4","Type":"ContainerStarted","Data":"184c6e2e5ffb159849fc19bb80e2f772872e90300a27f4117924d3d34d1941c3"}
Oct 30 05:14:53.388485 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:53.388115    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8afb26-688849cfdf-6b6c5" event={"ID":"96952584-fdf3-4731-a845-83ec28cd69b4","Type":"ContainerStarted","Data":"f889aa3073e97f7404cf8ffa328de86fb5dd9f3fd441631300228c0930b36c91"}
Oct 30 05:14:53.388485 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:53.388244    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8afb26-688849cfdf-6b6c5"
Oct 30 05:14:53.431562 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:53.431508    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8afb26-688849cfdf-6b6c5" podStartSLOduration=2.055535607 podStartE2EDuration="2.431487516s" totalImagesPullingTime="375.951909ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:14:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:14:53.429786511 +0000 UTC m=+1822.653606442" watchObservedRunningTime="2026-10-30 05:14:53.431487516 +0000 UTC m=+1822.655307418"
Oct 30 05:14:55.176416 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:55.176383    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-c7f962/test-model-c7f962-kserve-5fd5595fd9-jbf8l"
Oct 30 05:14:55.194649 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:55.194619    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-c7f962/test-model-c7f962-kserve-5fd5595fd9-jbf8l"
Oct 30 05:14:55.296605 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:55.296553    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:55.416892 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:55.416859    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-c7f962/test-model-c7f962-kserve-5fd5595fd9-jbf8l"
Oct 30 05:14:56.837997 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:14:56.837957    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:14:56.850005 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:56.849976    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:57.062924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:57.053394    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:57.190403 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:57.190317    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:57.396397 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:57.396350    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:57.561569 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:57.561478    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:57.690768 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:57.690730    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:58.269618 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:58.269589    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:14:59.400461 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:59.400431    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8afb26-688849cfdf-6b6c5"
Oct 30 05:14:59.799340 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:14:59.799251    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:00.209726 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.209669    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888955-5v2p9"]
Oct 30 05:15:00.214371 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.214346    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888955-5v2p9"
Oct 30 05:15:00.217200 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.217175    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c3486b-29888955-2n6bs"]
Oct 30 05:15:00.221631 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.221607    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c3486b-29888955-2n6bs"
Oct 30 05:15:00.222679 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.222649    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888955-x2zzg"]
Oct 30 05:15:00.227661 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.227638    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888955-x2zzg"
Oct 30 05:15:00.243174 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.243127    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888955-5v2p9"]
Oct 30 05:15:00.293535 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.293508    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\""
Oct 30 05:15:00.306801 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.306763    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c3486b-29888955-2n6bs"]
Oct 30 05:15:00.312244 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.312217    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888955-x2zzg"]
Oct 30 05:15:00.316414 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.316385    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888955-zvsnz"]
Oct 30 05:15:00.323188 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.320314    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b453b1b2-fd20-4f49-ba72-9cc2588e4201-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888955-5v2p9\" (UID: \"b453b1b2-fd20-4f49-ba72-9cc2588e4201\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888955-5v2p9"
Oct 30 05:15:00.323319 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.323208    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-297z9\" (UniqueName: \"kubernetes.io/projected/b453b1b2-fd20-4f49-ba72-9cc2588e4201-kube-api-access-297z9\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888955-5v2p9\" (UID: \"b453b1b2-fd20-4f49-ba72-9cc2588e4201\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888955-5v2p9"
Oct 30 05:15:00.323319 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.323314    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l6tx\" (UniqueName: \"kubernetes.io/projected/d8920d90-c187-4e68-a3a0-ff76a5f28d1b-kube-api-access-7l6tx\") pod \"maas-api-key-cleanup-e2e-api-b-c3486b-29888955-2n6bs\" (UID: \"d8920d90-c187-4e68-a3a0-ff76a5f28d1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c3486b-29888955-2n6bs"
Oct 30 05:15:00.323610 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.323368    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71250a09-9976-4049-9280-de3ccdfd3ab0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888955-x2zzg\" (UID: \"71250a09-9976-4049-9280-de3ccdfd3ab0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888955-x2zzg"
Oct 30 05:15:00.323610 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.323415    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8920d90-c187-4e68-a3a0-ff76a5f28d1b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-c3486b-29888955-2n6bs\" (UID: \"d8920d90-c187-4e68-a3a0-ff76a5f28d1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c3486b-29888955-2n6bs"
Oct 30 05:15:00.323610 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.323480    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbtv4\" (UniqueName: \"kubernetes.io/projected/71250a09-9976-4049-9280-de3ccdfd3ab0-kube-api-access-lbtv4\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888955-x2zzg\" (UID: \"71250a09-9976-4049-9280-de3ccdfd3ab0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888955-x2zzg"
Oct 30 05:15:00.323789 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.322297    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-010013-29888955-d4m2d"]
Oct 30 05:15:00.325011 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.322389    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888955-zvsnz"
Oct 30 05:15:00.330664 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.330520    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ccafe0-29888955-mh52k"]
Oct 30 05:15:00.333597 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.333267    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-010013-29888955-d4m2d"
Oct 30 05:15:00.354792 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.354092    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888955-zvsnz"]
Oct 30 05:15:00.354792 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.354205    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ccafe0-29888955-mh52k"
Oct 30 05:15:00.356073 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.356052    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-010013-29888955-d4m2d"]
Oct 30 05:15:00.361112 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.361091    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ccafe0-29888955-mh52k"]
Oct 30 05:15:00.367344 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.367326    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888955-gw8mj"]
Oct 30 05:15:00.372510 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.372479    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888955-5njn6"]
Oct 30 05:15:00.372622 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.372561    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888955-gw8mj"
Oct 30 05:15:00.376680 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.376657    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888955-5njn6"
Oct 30 05:15:00.376851 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.376834    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-36fd3c-29888955-t42l4"]
Oct 30 05:15:00.382060 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.382039    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-36fd3c-29888955-t42l4"
Oct 30 05:15:00.386082 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.386057    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888955-gw8mj"]
Oct 30 05:15:00.389103 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.389081    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-36fd3c-29888955-t42l4"]
Oct 30 05:15:00.393124 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.393106    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888955-5njn6"]
Oct 30 05:15:00.425097 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.425069    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/206ca9bd-dbed-464d-9bff-4a6195a3cda9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-83e83e-29888955-zvsnz\" (UID: \"206ca9bd-dbed-464d-9bff-4a6195a3cda9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888955-zvsnz"
Oct 30 05:15:00.425479 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.425110    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb3a1f87-17d3-42d5-9019-78a8b6d48825-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-36fd3c-29888955-t42l4\" (UID: \"bb3a1f87-17d3-42d5-9019-78a8b6d48825\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-36fd3c-29888955-t42l4"
Oct 30 05:15:00.425479 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.425135    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b453b1b2-fd20-4f49-ba72-9cc2588e4201-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888955-5v2p9\" (UID: \"b453b1b2-fd20-4f49-ba72-9cc2588e4201\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888955-5v2p9"
Oct 30 05:15:00.425479 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.425239    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-297z9\" (UniqueName: \"kubernetes.io/projected/b453b1b2-fd20-4f49-ba72-9cc2588e4201-kube-api-access-297z9\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888955-5v2p9\" (UID: \"b453b1b2-fd20-4f49-ba72-9cc2588e4201\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888955-5v2p9"
Oct 30 05:15:00.425479 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.425350    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgjkk\" (UniqueName: \"kubernetes.io/projected/4bc6a4c7-1a9d-4f6e-bf5b-03138318b1c8-kube-api-access-cgjkk\") pod \"maas-api-key-cleanup-e2e-inf-a-010013-29888955-d4m2d\" (UID: \"4bc6a4c7-1a9d-4f6e-bf5b-03138318b1c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-010013-29888955-d4m2d"
Oct 30 05:15:00.425479 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.425389    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l6tx\" (UniqueName: \"kubernetes.io/projected/d8920d90-c187-4e68-a3a0-ff76a5f28d1b-kube-api-access-7l6tx\") pod \"maas-api-key-cleanup-e2e-api-b-c3486b-29888955-2n6bs\" (UID: \"d8920d90-c187-4e68-a3a0-ff76a5f28d1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c3486b-29888955-2n6bs"
Oct 30 05:15:00.425479 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.425461    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71250a09-9976-4049-9280-de3ccdfd3ab0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888955-x2zzg\" (UID: \"71250a09-9976-4049-9280-de3ccdfd3ab0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888955-x2zzg"
Oct 30 05:15:00.425811 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.425502    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ee1e0d5-861e-41ac-929d-cfbdb7d75cd6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-ccafe0-29888955-mh52k\" (UID: \"3ee1e0d5-861e-41ac-929d-cfbdb7d75cd6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ccafe0-29888955-mh52k"
Oct 30 05:15:00.425811 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.425524    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx7rv\" (UniqueName: \"kubernetes.io/projected/206ca9bd-dbed-464d-9bff-4a6195a3cda9-kube-api-access-lx7rv\") pod \"maas-api-key-cleanup-e2e-worker-w6-83e83e-29888955-zvsnz\" (UID: \"206ca9bd-dbed-464d-9bff-4a6195a3cda9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888955-zvsnz"
Oct 30 05:15:00.425811 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.425551    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8920d90-c187-4e68-a3a0-ff76a5f28d1b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-c3486b-29888955-2n6bs\" (UID: \"d8920d90-c187-4e68-a3a0-ff76a5f28d1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c3486b-29888955-2n6bs"
Oct 30 05:15:00.425811 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.425626    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bc6a4c7-1a9d-4f6e-bf5b-03138318b1c8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-010013-29888955-d4m2d\" (UID: \"4bc6a4c7-1a9d-4f6e-bf5b-03138318b1c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-010013-29888955-d4m2d"
Oct 30 05:15:00.425811 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.425671    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/938a8472-fb50-452a-bef7-416b671f3362-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888955-gw8mj\" (UID: \"938a8472-fb50-452a-bef7-416b671f3362\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888955-gw8mj"
Oct 30 05:15:00.425811 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.425713    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbtv4\" (UniqueName: \"kubernetes.io/projected/71250a09-9976-4049-9280-de3ccdfd3ab0-kube-api-access-lbtv4\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888955-x2zzg\" (UID: \"71250a09-9976-4049-9280-de3ccdfd3ab0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888955-x2zzg"
Oct 30 05:15:00.425811 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.425726    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b453b1b2-fd20-4f49-ba72-9cc2588e4201-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888955-5v2p9\" (UID: \"b453b1b2-fd20-4f49-ba72-9cc2588e4201\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888955-5v2p9"
Oct 30 05:15:00.426234 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.425851    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5hld\" (UniqueName: \"kubernetes.io/projected/938a8472-fb50-452a-bef7-416b671f3362-kube-api-access-w5hld\") pod \"maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888955-gw8mj\" (UID: \"938a8472-fb50-452a-bef7-416b671f3362\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888955-gw8mj"
Oct 30 05:15:00.426234 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.425890    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fh6d\" (UniqueName: \"kubernetes.io/projected/3ee1e0d5-861e-41ac-929d-cfbdb7d75cd6-kube-api-access-5fh6d\") pod \"maas-api-key-cleanup-e2e-inf-a-ccafe0-29888955-mh52k\" (UID: \"3ee1e0d5-861e-41ac-929d-cfbdb7d75cd6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ccafe0-29888955-mh52k"
Oct 30 05:15:00.426234 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.426063    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8pk9\" (UniqueName: \"kubernetes.io/projected/88933a2c-374f-4eb4-98ed-b7f3438774c3-kube-api-access-w8pk9\") pod \"maas-api-key-cleanup-e2e-worker-w0-b21507-29888955-5njn6\" (UID: \"88933a2c-374f-4eb4-98ed-b7f3438774c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888955-5njn6"
Oct 30 05:15:00.426234 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.426111    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g77m9\" (UniqueName: \"kubernetes.io/projected/bb3a1f87-17d3-42d5-9019-78a8b6d48825-kube-api-access-g77m9\") pod \"maas-api-key-cleanup-e2e-inf-b-36fd3c-29888955-t42l4\" (UID: \"bb3a1f87-17d3-42d5-9019-78a8b6d48825\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-36fd3c-29888955-t42l4"
Oct 30 05:15:00.426234 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.426150    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88933a2c-374f-4eb4-98ed-b7f3438774c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b21507-29888955-5njn6\" (UID: \"88933a2c-374f-4eb4-98ed-b7f3438774c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888955-5njn6"
Oct 30 05:15:00.426234 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.426163    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8920d90-c187-4e68-a3a0-ff76a5f28d1b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-c3486b-29888955-2n6bs\" (UID: \"d8920d90-c187-4e68-a3a0-ff76a5f28d1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c3486b-29888955-2n6bs"
Oct 30 05:15:00.426477 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.426264    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71250a09-9976-4049-9280-de3ccdfd3ab0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888955-x2zzg\" (UID: \"71250a09-9976-4049-9280-de3ccdfd3ab0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888955-x2zzg"
Oct 30 05:15:00.446351 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.446326    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbtv4\" (UniqueName: \"kubernetes.io/projected/71250a09-9976-4049-9280-de3ccdfd3ab0-kube-api-access-lbtv4\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888955-x2zzg\" (UID: \"71250a09-9976-4049-9280-de3ccdfd3ab0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888955-x2zzg"
Oct 30 05:15:00.446450 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.446331    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l6tx\" (UniqueName: \"kubernetes.io/projected/d8920d90-c187-4e68-a3a0-ff76a5f28d1b-kube-api-access-7l6tx\") pod \"maas-api-key-cleanup-e2e-api-b-c3486b-29888955-2n6bs\" (UID: \"d8920d90-c187-4e68-a3a0-ff76a5f28d1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c3486b-29888955-2n6bs"
Oct 30 05:15:00.446612 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.446590    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-297z9\" (UniqueName: \"kubernetes.io/projected/b453b1b2-fd20-4f49-ba72-9cc2588e4201-kube-api-access-297z9\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888955-5v2p9\" (UID: \"b453b1b2-fd20-4f49-ba72-9cc2588e4201\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888955-5v2p9"
Oct 30 05:15:00.527850 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.527759    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ee1e0d5-861e-41ac-929d-cfbdb7d75cd6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-ccafe0-29888955-mh52k\" (UID: \"3ee1e0d5-861e-41ac-929d-cfbdb7d75cd6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ccafe0-29888955-mh52k"
Oct 30 05:15:00.527850 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.527801    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx7rv\" (UniqueName: \"kubernetes.io/projected/206ca9bd-dbed-464d-9bff-4a6195a3cda9-kube-api-access-lx7rv\") pod \"maas-api-key-cleanup-e2e-worker-w6-83e83e-29888955-zvsnz\" (UID: \"206ca9bd-dbed-464d-9bff-4a6195a3cda9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888955-zvsnz"
Oct 30 05:15:00.527850 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.527841    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bc6a4c7-1a9d-4f6e-bf5b-03138318b1c8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-010013-29888955-d4m2d\" (UID: \"4bc6a4c7-1a9d-4f6e-bf5b-03138318b1c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-010013-29888955-d4m2d"
Oct 30 05:15:00.528157 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.527869    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/938a8472-fb50-452a-bef7-416b671f3362-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888955-gw8mj\" (UID: \"938a8472-fb50-452a-bef7-416b671f3362\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888955-gw8mj"
Oct 30 05:15:00.528157 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.527983    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5hld\" (UniqueName: \"kubernetes.io/projected/938a8472-fb50-452a-bef7-416b671f3362-kube-api-access-w5hld\") pod \"maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888955-gw8mj\" (UID: \"938a8472-fb50-452a-bef7-416b671f3362\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888955-gw8mj"
Oct 30 05:15:00.528157 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.528014    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fh6d\" (UniqueName: \"kubernetes.io/projected/3ee1e0d5-861e-41ac-929d-cfbdb7d75cd6-kube-api-access-5fh6d\") pod \"maas-api-key-cleanup-e2e-inf-a-ccafe0-29888955-mh52k\" (UID: \"3ee1e0d5-861e-41ac-929d-cfbdb7d75cd6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ccafe0-29888955-mh52k"
Oct 30 05:15:00.528157 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.528098    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8pk9\" (UniqueName: \"kubernetes.io/projected/88933a2c-374f-4eb4-98ed-b7f3438774c3-kube-api-access-w8pk9\") pod \"maas-api-key-cleanup-e2e-worker-w0-b21507-29888955-5njn6\" (UID: \"88933a2c-374f-4eb4-98ed-b7f3438774c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888955-5njn6"
Oct 30 05:15:00.528157 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.528124    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g77m9\" (UniqueName: \"kubernetes.io/projected/bb3a1f87-17d3-42d5-9019-78a8b6d48825-kube-api-access-g77m9\") pod \"maas-api-key-cleanup-e2e-inf-b-36fd3c-29888955-t42l4\" (UID: \"bb3a1f87-17d3-42d5-9019-78a8b6d48825\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-36fd3c-29888955-t42l4"
Oct 30 05:15:00.528405 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.528174    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88933a2c-374f-4eb4-98ed-b7f3438774c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b21507-29888955-5njn6\" (UID: \"88933a2c-374f-4eb4-98ed-b7f3438774c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888955-5njn6"
Oct 30 05:15:00.528405 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.528250    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/206ca9bd-dbed-464d-9bff-4a6195a3cda9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-83e83e-29888955-zvsnz\" (UID: \"206ca9bd-dbed-464d-9bff-4a6195a3cda9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888955-zvsnz"
Oct 30 05:15:00.528405 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.528295    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb3a1f87-17d3-42d5-9019-78a8b6d48825-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-36fd3c-29888955-t42l4\" (UID: \"bb3a1f87-17d3-42d5-9019-78a8b6d48825\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-36fd3c-29888955-t42l4"
Oct 30 05:15:00.528624 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.528408    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgjkk\" (UniqueName: \"kubernetes.io/projected/4bc6a4c7-1a9d-4f6e-bf5b-03138318b1c8-kube-api-access-cgjkk\") pod \"maas-api-key-cleanup-e2e-inf-a-010013-29888955-d4m2d\" (UID: \"4bc6a4c7-1a9d-4f6e-bf5b-03138318b1c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-010013-29888955-d4m2d"
Oct 30 05:15:00.528624 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.528592    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ee1e0d5-861e-41ac-929d-cfbdb7d75cd6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-ccafe0-29888955-mh52k\" (UID: \"3ee1e0d5-861e-41ac-929d-cfbdb7d75cd6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ccafe0-29888955-mh52k"
Oct 30 05:15:00.528737 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.528670    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/938a8472-fb50-452a-bef7-416b671f3362-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888955-gw8mj\" (UID: \"938a8472-fb50-452a-bef7-416b671f3362\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888955-gw8mj"
Oct 30 05:15:00.528737 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.528695    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bc6a4c7-1a9d-4f6e-bf5b-03138318b1c8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-010013-29888955-d4m2d\" (UID: \"4bc6a4c7-1a9d-4f6e-bf5b-03138318b1c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-010013-29888955-d4m2d"
Oct 30 05:15:00.529288 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.529266    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/206ca9bd-dbed-464d-9bff-4a6195a3cda9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-83e83e-29888955-zvsnz\" (UID: \"206ca9bd-dbed-464d-9bff-4a6195a3cda9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888955-zvsnz"
Oct 30 05:15:00.529397 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.529376    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88933a2c-374f-4eb4-98ed-b7f3438774c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b21507-29888955-5njn6\" (UID: \"88933a2c-374f-4eb4-98ed-b7f3438774c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888955-5njn6"
Oct 30 05:15:00.529435 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.529418    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb3a1f87-17d3-42d5-9019-78a8b6d48825-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-36fd3c-29888955-t42l4\" (UID: \"bb3a1f87-17d3-42d5-9019-78a8b6d48825\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-36fd3c-29888955-t42l4"
Oct 30 05:15:00.548281 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.548257    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5hld\" (UniqueName: \"kubernetes.io/projected/938a8472-fb50-452a-bef7-416b671f3362-kube-api-access-w5hld\") pod \"maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888955-gw8mj\" (UID: \"938a8472-fb50-452a-bef7-416b671f3362\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888955-gw8mj"
Oct 30 05:15:00.552877 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.552854    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx7rv\" (UniqueName: \"kubernetes.io/projected/206ca9bd-dbed-464d-9bff-4a6195a3cda9-kube-api-access-lx7rv\") pod \"maas-api-key-cleanup-e2e-worker-w6-83e83e-29888955-zvsnz\" (UID: \"206ca9bd-dbed-464d-9bff-4a6195a3cda9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888955-zvsnz"
Oct 30 05:15:00.554636 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.554608    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fh6d\" (UniqueName: \"kubernetes.io/projected/3ee1e0d5-861e-41ac-929d-cfbdb7d75cd6-kube-api-access-5fh6d\") pod \"maas-api-key-cleanup-e2e-inf-a-ccafe0-29888955-mh52k\" (UID: \"3ee1e0d5-861e-41ac-929d-cfbdb7d75cd6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ccafe0-29888955-mh52k"
Oct 30 05:15:00.554747 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.554725    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g77m9\" (UniqueName: \"kubernetes.io/projected/bb3a1f87-17d3-42d5-9019-78a8b6d48825-kube-api-access-g77m9\") pod \"maas-api-key-cleanup-e2e-inf-b-36fd3c-29888955-t42l4\" (UID: \"bb3a1f87-17d3-42d5-9019-78a8b6d48825\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-36fd3c-29888955-t42l4"
Oct 30 05:15:00.554914 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.554880    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8pk9\" (UniqueName: \"kubernetes.io/projected/88933a2c-374f-4eb4-98ed-b7f3438774c3-kube-api-access-w8pk9\") pod \"maas-api-key-cleanup-e2e-worker-w0-b21507-29888955-5njn6\" (UID: \"88933a2c-374f-4eb4-98ed-b7f3438774c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888955-5njn6"
Oct 30 05:15:00.555295 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.555273    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgjkk\" (UniqueName: \"kubernetes.io/projected/4bc6a4c7-1a9d-4f6e-bf5b-03138318b1c8-kube-api-access-cgjkk\") pod \"maas-api-key-cleanup-e2e-inf-a-010013-29888955-d4m2d\" (UID: \"4bc6a4c7-1a9d-4f6e-bf5b-03138318b1c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-010013-29888955-d4m2d"
Oct 30 05:15:00.557084 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.557062    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888955-5v2p9"
Oct 30 05:15:00.567800 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.567771    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c3486b-29888955-2n6bs"
Oct 30 05:15:00.579672 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.579638    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888955-x2zzg"
Oct 30 05:15:00.658815 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.658500    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888955-zvsnz"
Oct 30 05:15:00.668386 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.668176    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-010013-29888955-d4m2d"
Oct 30 05:15:00.676338 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.676318    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ccafe0-29888955-mh52k"
Oct 30 05:15:00.693516 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.693478    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888955-gw8mj"
Oct 30 05:15:00.700510 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.700486    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888955-5njn6"
Oct 30 05:15:00.708210 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.708183    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-36fd3c-29888955-t42l4"
Oct 30 05:15:00.882170 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.882138    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888955-5v2p9"]
Oct 30 05:15:00.911481 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:00.911186    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c3486b-29888955-2n6bs"]
Oct 30 05:15:01.004296 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:01.004264    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-010013-29888955-d4m2d"]
Oct 30 05:15:01.054492 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:01.054455    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ccafe0-29888955-mh52k"]
Oct 30 05:15:01.078753 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:15:01.078686    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bc6a4c7_1a9d_4f6e_bf5b_03138318b1c8.slice/crio-dbcce354f2215bb9664655e4099ccd5ce8b67d08cfdc0ba7b1fa0ff82c019d2d WatchSource:0}: Error finding container dbcce354f2215bb9664655e4099ccd5ce8b67d08cfdc0ba7b1fa0ff82c019d2d: Status 404 returned error can't find the container with id dbcce354f2215bb9664655e4099ccd5ce8b67d08cfdc0ba7b1fa0ff82c019d2d
Oct 30 05:15:01.080738 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:15:01.080695    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ee1e0d5_861e_41ac_929d_cfbdb7d75cd6.slice/crio-ee2b85897072cdc3afd46644396bc90458389d9e4ce96df50922f62a7e2dc9fe WatchSource:0}: Error finding container ee2b85897072cdc3afd46644396bc90458389d9e4ce96df50922f62a7e2dc9fe: Status 404 returned error can't find the container with id ee2b85897072cdc3afd46644396bc90458389d9e4ce96df50922f62a7e2dc9fe
Oct 30 05:15:01.116125 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:01.116093    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888955-gw8mj"]
Oct 30 05:15:01.116754 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:15:01.116731    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod938a8472_fb50_452a_bef7_416b671f3362.slice/crio-394342e6e1c2f5ed539d8780c928fcbd4dfc70720e33cb488d4b12afe532b4bf WatchSource:0}: Error finding container 394342e6e1c2f5ed539d8780c928fcbd4dfc70720e33cb488d4b12afe532b4bf: Status 404 returned error can't find the container with id 394342e6e1c2f5ed539d8780c928fcbd4dfc70720e33cb488d4b12afe532b4bf
Oct 30 05:15:01.120557 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:15:01.120008    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71250a09_9976_4049_9280_de3ccdfd3ab0.slice/crio-bd56d68a831e4947e2a464dd41dec09d59129d96e1fd7af581a47fb5c97b74cd WatchSource:0}: Error finding container bd56d68a831e4947e2a464dd41dec09d59129d96e1fd7af581a47fb5c97b74cd: Status 404 returned error can't find the container with id bd56d68a831e4947e2a464dd41dec09d59129d96e1fd7af581a47fb5c97b74cd
Oct 30 05:15:01.121309 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:01.121289    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888955-x2zzg"]
Oct 30 05:15:01.126078 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:01.125694    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-36fd3c-29888955-t42l4"]
Oct 30 05:15:01.131015 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:01.130731    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888955-5njn6"]
Oct 30 05:15:01.203247 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:01.203222    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888955-zvsnz"]
Oct 30 05:15:01.379374 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:15:01.379345    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod206ca9bd_dbed_464d_9bff_4a6195a3cda9.slice/crio-788d8b31b96384a728fb4b5f0ca5411675128561f9247fe1b215e50e0a50eba9 WatchSource:0}: Error finding container 788d8b31b96384a728fb4b5f0ca5411675128561f9247fe1b215e50e0a50eba9: Status 404 returned error can't find the container with id 788d8b31b96384a728fb4b5f0ca5411675128561f9247fe1b215e50e0a50eba9
Oct 30 05:15:01.484927 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:01.484878    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888955-5njn6" event={"ID":"88933a2c-374f-4eb4-98ed-b7f3438774c3","Type":"ContainerStarted","Data":"8f41f38571d67e130334c9799b10caff28096a06824e05486c6e070f61347020"}
Oct 30 05:15:01.486346 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:01.486312    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ccafe0-29888955-mh52k" event={"ID":"3ee1e0d5-861e-41ac-929d-cfbdb7d75cd6","Type":"ContainerStarted","Data":"ee2b85897072cdc3afd46644396bc90458389d9e4ce96df50922f62a7e2dc9fe"}
Oct 30 05:15:01.487696 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:01.487671    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888955-5v2p9" event={"ID":"b453b1b2-fd20-4f49-ba72-9cc2588e4201","Type":"ContainerStarted","Data":"acf7131df3e1288c21d8835a26cdce8c86ef6587a99379cb304c5d09e70279fd"}
Oct 30 05:15:01.488920 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:01.488879    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888955-x2zzg" event={"ID":"71250a09-9976-4049-9280-de3ccdfd3ab0","Type":"ContainerStarted","Data":"bd56d68a831e4947e2a464dd41dec09d59129d96e1fd7af581a47fb5c97b74cd"}
Oct 30 05:15:01.490035 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:01.490015    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888955-zvsnz" event={"ID":"206ca9bd-dbed-464d-9bff-4a6195a3cda9","Type":"ContainerStarted","Data":"788d8b31b96384a728fb4b5f0ca5411675128561f9247fe1b215e50e0a50eba9"}
Oct 30 05:15:01.491076 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:01.491054    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888955-gw8mj" event={"ID":"938a8472-fb50-452a-bef7-416b671f3362","Type":"ContainerStarted","Data":"394342e6e1c2f5ed539d8780c928fcbd4dfc70720e33cb488d4b12afe532b4bf"}
Oct 30 05:15:01.492126 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:01.492105    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-36fd3c-29888955-t42l4" event={"ID":"bb3a1f87-17d3-42d5-9019-78a8b6d48825","Type":"ContainerStarted","Data":"c2cf763ca81344a8e61ef7d5240614716c3de14eae39242a49546b4ecc8eee62"}
Oct 30 05:15:01.493149 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:01.493129    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-010013-29888955-d4m2d" event={"ID":"4bc6a4c7-1a9d-4f6e-bf5b-03138318b1c8","Type":"ContainerStarted","Data":"dbcce354f2215bb9664655e4099ccd5ce8b67d08cfdc0ba7b1fa0ff82c019d2d"}
Oct 30 05:15:01.494189 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:01.494170    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c3486b-29888955-2n6bs" event={"ID":"d8920d90-c187-4e68-a3a0-ff76a5f28d1b","Type":"ContainerStarted","Data":"30cbe54bbb7c0e7db064bf0f7036e2b8b5bd2a8837ada5459363e47ae36ab6a1"}
Oct 30 05:15:01.959599 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:01.959463    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:02.503225 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:02.503192    5898 generic.go:356] "Generic (PLEG): container finished" podID="71250a09-9976-4049-9280-de3ccdfd3ab0" containerID="4a100c31dc26c634b99c1d4fe088f10199bf70ede9bb6abe4fd53eb6b692b9b9" exitCode=60
Oct 30 05:15:02.503582 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:02.503258    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888955-x2zzg" event={"ID":"71250a09-9976-4049-9280-de3ccdfd3ab0","Type":"ContainerDied","Data":"4a100c31dc26c634b99c1d4fe088f10199bf70ede9bb6abe4fd53eb6b692b9b9"}
Oct 30 05:15:02.503636 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:02.503605    5898 scope.go:122] "RemoveContainer" containerID="4a100c31dc26c634b99c1d4fe088f10199bf70ede9bb6abe4fd53eb6b692b9b9"
Oct 30 05:15:02.505098 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:02.505065    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888955-zvsnz" event={"ID":"206ca9bd-dbed-464d-9bff-4a6195a3cda9","Type":"ContainerStarted","Data":"7a71404924b1cd2a4d42b0610407d0f363b2e4d9acd1566b688c7a0060ffabfc"}
Oct 30 05:15:02.506817 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:02.506794    5898 generic.go:356] "Generic (PLEG): container finished" podID="938a8472-fb50-452a-bef7-416b671f3362" containerID="684ee8eab94574480facea2834cafa3ed8435969e140549eb0bb92ce90862f16" exitCode=60
Oct 30 05:15:02.506935 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:02.506872    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888955-gw8mj" event={"ID":"938a8472-fb50-452a-bef7-416b671f3362","Type":"ContainerDied","Data":"684ee8eab94574480facea2834cafa3ed8435969e140549eb0bb92ce90862f16"}
Oct 30 05:15:02.507169 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:02.507149    5898 scope.go:122] "RemoveContainer" containerID="684ee8eab94574480facea2834cafa3ed8435969e140549eb0bb92ce90862f16"
Oct 30 05:15:02.508555 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:02.508530    5898 generic.go:356] "Generic (PLEG): container finished" podID="bb3a1f87-17d3-42d5-9019-78a8b6d48825" containerID="8282141856d66166276369bc3297b1ed07fdad70f2abdf25e42ccf0025382778" exitCode=60
Oct 30 05:15:02.508655 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:02.508594    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-36fd3c-29888955-t42l4" event={"ID":"bb3a1f87-17d3-42d5-9019-78a8b6d48825","Type":"ContainerDied","Data":"8282141856d66166276369bc3297b1ed07fdad70f2abdf25e42ccf0025382778"}
Oct 30 05:15:02.508856 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:02.508839    5898 scope.go:122] "RemoveContainer" containerID="8282141856d66166276369bc3297b1ed07fdad70f2abdf25e42ccf0025382778"
Oct 30 05:15:02.510195 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:02.510179    5898 generic.go:356] "Generic (PLEG): container finished" podID="4bc6a4c7-1a9d-4f6e-bf5b-03138318b1c8" containerID="25588a28f7f42e424ab29f4aeef8c83e4cb48eab297a249d4aede261bb3e4c1a" exitCode=60
Oct 30 05:15:02.510277 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:02.510242    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-010013-29888955-d4m2d" event={"ID":"4bc6a4c7-1a9d-4f6e-bf5b-03138318b1c8","Type":"ContainerDied","Data":"25588a28f7f42e424ab29f4aeef8c83e4cb48eab297a249d4aede261bb3e4c1a"}
Oct 30 05:15:02.510496 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:02.510479    5898 scope.go:122] "RemoveContainer" containerID="25588a28f7f42e424ab29f4aeef8c83e4cb48eab297a249d4aede261bb3e4c1a"
Oct 30 05:15:02.511710 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:02.511694    5898 generic.go:356] "Generic (PLEG): container finished" podID="d8920d90-c187-4e68-a3a0-ff76a5f28d1b" containerID="fcce91d091e8963bccd44af6b473950dcdcc899d17423441c5901acd6df52ed4" exitCode=60
Oct 30 05:15:02.511808 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:02.511772    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c3486b-29888955-2n6bs" event={"ID":"d8920d90-c187-4e68-a3a0-ff76a5f28d1b","Type":"ContainerDied","Data":"fcce91d091e8963bccd44af6b473950dcdcc899d17423441c5901acd6df52ed4"}
Oct 30 05:15:02.512045 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:02.512028    5898 scope.go:122] "RemoveContainer" containerID="fcce91d091e8963bccd44af6b473950dcdcc899d17423441c5901acd6df52ed4"
Oct 30 05:15:02.519764 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:02.519736    5898 generic.go:356] "Generic (PLEG): container finished" podID="88933a2c-374f-4eb4-98ed-b7f3438774c3" containerID="975cb3fbc7eef5408534aeffef65bc831e21cc22d206b3bd887a94808ef38ae4" exitCode=60
Oct 30 05:15:02.519877 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:02.519816    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888955-5njn6" event={"ID":"88933a2c-374f-4eb4-98ed-b7f3438774c3","Type":"ContainerDied","Data":"975cb3fbc7eef5408534aeffef65bc831e21cc22d206b3bd887a94808ef38ae4"}
Oct 30 05:15:02.520091 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:02.520073    5898 scope.go:122] "RemoveContainer" containerID="975cb3fbc7eef5408534aeffef65bc831e21cc22d206b3bd887a94808ef38ae4"
Oct 30 05:15:02.522024 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:02.522006    5898 generic.go:356] "Generic (PLEG): container finished" podID="3ee1e0d5-861e-41ac-929d-cfbdb7d75cd6" containerID="007abadfbf113d9cacf7564f129d77f21276ac4ce3dbb8d828331302f4bc2e21" exitCode=60
Oct 30 05:15:02.522137 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:02.522072    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ccafe0-29888955-mh52k" event={"ID":"3ee1e0d5-861e-41ac-929d-cfbdb7d75cd6","Type":"ContainerDied","Data":"007abadfbf113d9cacf7564f129d77f21276ac4ce3dbb8d828331302f4bc2e21"}
Oct 30 05:15:02.522336 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:02.522306    5898 scope.go:122] "RemoveContainer" containerID="007abadfbf113d9cacf7564f129d77f21276ac4ce3dbb8d828331302f4bc2e21"
Oct 30 05:15:02.524783 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:02.524763    5898 generic.go:356] "Generic (PLEG): container finished" podID="b453b1b2-fd20-4f49-ba72-9cc2588e4201" containerID="cd338bc0a223171a46a3ab39b77e4807ff01364766c0c84e646e14fbab280936" exitCode=60
Oct 30 05:15:02.525050 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:02.524831    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888955-5v2p9" event={"ID":"b453b1b2-fd20-4f49-ba72-9cc2588e4201","Type":"ContainerDied","Data":"cd338bc0a223171a46a3ab39b77e4807ff01364766c0c84e646e14fbab280936"}
Oct 30 05:15:02.525263 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:02.525083    5898 scope.go:122] "RemoveContainer" containerID="cd338bc0a223171a46a3ab39b77e4807ff01364766c0c84e646e14fbab280936"
Oct 30 05:15:02.739619 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:02.739564    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888955-zvsnz" podStartSLOduration=2.73955111 podStartE2EDuration="2.73955111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:15:02.736211416 +0000 UTC m=+1831.960031330" watchObservedRunningTime="2026-10-30 05:15:02.73955111 +0000 UTC m=+1831.963371057"
Oct 30 05:15:03.534539 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:03.534510    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888955-x2zzg" event={"ID":"71250a09-9976-4049-9280-de3ccdfd3ab0","Type":"ContainerStarted","Data":"4f36755d68d5c1b9ebeeb62c3498d4731e2efbbc27e8ab5792917dc3520c5702"}
Oct 30 05:15:03.536084 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:03.536061    5898 generic.go:356] "Generic (PLEG): container finished" podID="206ca9bd-dbed-464d-9bff-4a6195a3cda9" containerID="7a71404924b1cd2a4d42b0610407d0f363b2e4d9acd1566b688c7a0060ffabfc" exitCode=60
Oct 30 05:15:03.536189 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:03.536125    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888955-zvsnz" event={"ID":"206ca9bd-dbed-464d-9bff-4a6195a3cda9","Type":"ContainerDied","Data":"7a71404924b1cd2a4d42b0610407d0f363b2e4d9acd1566b688c7a0060ffabfc"}
Oct 30 05:15:03.536454 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:03.536437    5898 scope.go:122] "RemoveContainer" containerID="7a71404924b1cd2a4d42b0610407d0f363b2e4d9acd1566b688c7a0060ffabfc"
Oct 30 05:15:03.538025 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:03.538001    5898 generic.go:356] "Generic (PLEG): container finished" podID="938a8472-fb50-452a-bef7-416b671f3362" containerID="6bdbfa9a81f1a748588262e10295ed98de15546750950da1860f057ca4413329" exitCode=60
Oct 30 05:15:03.538115 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:03.538067    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888955-gw8mj" event={"ID":"938a8472-fb50-452a-bef7-416b671f3362","Type":"ContainerDied","Data":"6bdbfa9a81f1a748588262e10295ed98de15546750950da1860f057ca4413329"}
Oct 30 05:15:03.538177 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:03.538137    5898 scope.go:122] "RemoveContainer" containerID="684ee8eab94574480facea2834cafa3ed8435969e140549eb0bb92ce90862f16"
Oct 30 05:15:03.538398 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:03.538377    5898 scope.go:122] "RemoveContainer" containerID="6bdbfa9a81f1a748588262e10295ed98de15546750950da1860f057ca4413329"
Oct 30 05:15:03.538745 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:15:03.538700    5898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888955-gw8mj_odh-ai-gateway-infra(938a8472-fb50-452a-bef7-416b671f3362)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888955-gw8mj" podUID="938a8472-fb50-452a-bef7-416b671f3362"
Oct 30 05:15:03.540288 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:03.540265    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-36fd3c-29888955-t42l4" event={"ID":"bb3a1f87-17d3-42d5-9019-78a8b6d48825","Type":"ContainerStarted","Data":"583e2cf90c371c32deaee99e660026e3cfc3be763b3cd046faeeb5abccdd3f53"}
Oct 30 05:15:03.542224 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:03.542202    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-010013-29888955-d4m2d" event={"ID":"4bc6a4c7-1a9d-4f6e-bf5b-03138318b1c8","Type":"ContainerStarted","Data":"f3592d03c5c21df834d17d39fd6805dd3a2a0c504cdb037d28a3605d207bfbd5"}
Oct 30 05:15:03.544338 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:03.544315    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c3486b-29888955-2n6bs" event={"ID":"d8920d90-c187-4e68-a3a0-ff76a5f28d1b","Type":"ContainerStarted","Data":"0c60d426f17dda3eedecf4140fc9243f794755a3757b68f9d26079bcc3507023"}
Oct 30 05:15:03.546695 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:03.546627    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888955-5njn6" event={"ID":"88933a2c-374f-4eb4-98ed-b7f3438774c3","Type":"ContainerStarted","Data":"e0b3a3706d000f663d93a5d42fa6e1ac526c958971287abd328f4d499e664bfb"}
Oct 30 05:15:03.548836 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:03.548814    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ccafe0-29888955-mh52k" event={"ID":"3ee1e0d5-861e-41ac-929d-cfbdb7d75cd6","Type":"ContainerStarted","Data":"702759b2104dc067a87eb06578211db81a4195213ec7ac7fe61b48514b684d30"}
Oct 30 05:15:03.550747 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:03.550725    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888955-5v2p9" event={"ID":"b453b1b2-fd20-4f49-ba72-9cc2588e4201","Type":"ContainerStarted","Data":"20fde13b30731c3b3fa74f8fb3017441c6e9b518bfd4d8f2c85ec06dc1729f55"}
Oct 30 05:15:03.582525 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:03.582470    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888955-x2zzg" podStartSLOduration=3.582451023 podStartE2EDuration="3.582451023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:15:03.573193785 +0000 UTC m=+1832.797013683" watchObservedRunningTime="2026-10-30 05:15:03.582451023 +0000 UTC m=+1832.806270922"
Oct 30 05:15:03.618149 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:03.618101    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c3486b-29888955-2n6bs" podStartSLOduration=3.618088562 podStartE2EDuration="3.618088562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:15:03.613401886 +0000 UTC m=+1832.837221798" watchObservedRunningTime="2026-10-30 05:15:03.618088562 +0000 UTC m=+1832.841908463"
Oct 30 05:15:03.740568 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:03.740473    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-010013-29888955-d4m2d" podStartSLOduration=3.740451376 podStartE2EDuration="3.740451376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:15:03.738542357 +0000 UTC m=+1832.962362278" watchObservedRunningTime="2026-10-30 05:15:03.740451376 +0000 UTC m=+1832.964271282"
Oct 30 05:15:03.779153 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:03.779099    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888955-5v2p9" podStartSLOduration=3.779082417 podStartE2EDuration="3.779082417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:15:03.776990097 +0000 UTC m=+1833.000810025" watchObservedRunningTime="2026-10-30 05:15:03.779082417 +0000 UTC m=+1833.002902320"
Oct 30 05:15:03.819429 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:03.819383    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888955-5njn6" podStartSLOduration=3.819369182 podStartE2EDuration="3.819369182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:15:03.815753799 +0000 UTC m=+1833.039573700" watchObservedRunningTime="2026-10-30 05:15:03.819369182 +0000 UTC m=+1833.043189082"
Oct 30 05:15:03.862095 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:03.862034    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ccafe0-29888955-mh52k" podStartSLOduration=3.862015938 podStartE2EDuration="3.862015938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:15:03.854035417 +0000 UTC m=+1833.077855319" watchObservedRunningTime="2026-10-30 05:15:03.862015938 +0000 UTC m=+1833.085835840"
Oct 30 05:15:03.898383 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:03.898304    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-36fd3c-29888955-t42l4" podStartSLOduration=3.898284834 podStartE2EDuration="3.898284834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:15:03.896141193 +0000 UTC m=+1833.119961123" watchObservedRunningTime="2026-10-30 05:15:03.898284834 +0000 UTC m=+1833.122104737"
Oct 30 05:15:03.909934 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:03.909892    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:04.560158 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:04.560120    5898 scope.go:122] "RemoveContainer" containerID="6bdbfa9a81f1a748588262e10295ed98de15546750950da1860f057ca4413329"
Oct 30 05:15:04.560574 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:15:04.560462    5898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888955-gw8mj_odh-ai-gateway-infra(938a8472-fb50-452a-bef7-416b671f3362)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888955-gw8mj" podUID="938a8472-fb50-452a-bef7-416b671f3362"
Oct 30 05:15:04.562061 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:04.562031    5898 generic.go:356] "Generic (PLEG): container finished" podID="bb3a1f87-17d3-42d5-9019-78a8b6d48825" containerID="583e2cf90c371c32deaee99e660026e3cfc3be763b3cd046faeeb5abccdd3f53" exitCode=60
Oct 30 05:15:04.562213 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:04.562108    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-36fd3c-29888955-t42l4" event={"ID":"bb3a1f87-17d3-42d5-9019-78a8b6d48825","Type":"ContainerDied","Data":"583e2cf90c371c32deaee99e660026e3cfc3be763b3cd046faeeb5abccdd3f53"}
Oct 30 05:15:04.562213 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:04.562184    5898 scope.go:122] "RemoveContainer" containerID="8282141856d66166276369bc3297b1ed07fdad70f2abdf25e42ccf0025382778"
Oct 30 05:15:04.562373 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:04.562354    5898 scope.go:122] "RemoveContainer" containerID="583e2cf90c371c32deaee99e660026e3cfc3be763b3cd046faeeb5abccdd3f53"
Oct 30 05:15:04.562748 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:15:04.562710    5898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-36fd3c-29888955-t42l4_odh-ai-gateway-infra(bb3a1f87-17d3-42d5-9019-78a8b6d48825)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-36fd3c-29888955-t42l4" podUID="bb3a1f87-17d3-42d5-9019-78a8b6d48825"
Oct 30 05:15:04.564300 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:04.564281    5898 generic.go:356] "Generic (PLEG): container finished" podID="4bc6a4c7-1a9d-4f6e-bf5b-03138318b1c8" containerID="f3592d03c5c21df834d17d39fd6805dd3a2a0c504cdb037d28a3605d207bfbd5" exitCode=60
Oct 30 05:15:04.564395 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:04.564370    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-010013-29888955-d4m2d" event={"ID":"4bc6a4c7-1a9d-4f6e-bf5b-03138318b1c8","Type":"ContainerDied","Data":"f3592d03c5c21df834d17d39fd6805dd3a2a0c504cdb037d28a3605d207bfbd5"}
Oct 30 05:15:04.564585 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:04.564566    5898 scope.go:122] "RemoveContainer" containerID="f3592d03c5c21df834d17d39fd6805dd3a2a0c504cdb037d28a3605d207bfbd5"
Oct 30 05:15:04.564893 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:15:04.564872    5898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-010013-29888955-d4m2d_odh-ai-gateway-infra(4bc6a4c7-1a9d-4f6e-bf5b-03138318b1c8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-010013-29888955-d4m2d" podUID="4bc6a4c7-1a9d-4f6e-bf5b-03138318b1c8"
Oct 30 05:15:04.566362 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:04.566338    5898 generic.go:356] "Generic (PLEG): container finished" podID="d8920d90-c187-4e68-a3a0-ff76a5f28d1b" containerID="0c60d426f17dda3eedecf4140fc9243f794755a3757b68f9d26079bcc3507023" exitCode=60
Oct 30 05:15:04.566463 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:04.566418    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c3486b-29888955-2n6bs" event={"ID":"d8920d90-c187-4e68-a3a0-ff76a5f28d1b","Type":"ContainerDied","Data":"0c60d426f17dda3eedecf4140fc9243f794755a3757b68f9d26079bcc3507023"}
Oct 30 05:15:04.566668 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:04.566654    5898 scope.go:122] "RemoveContainer" containerID="0c60d426f17dda3eedecf4140fc9243f794755a3757b68f9d26079bcc3507023"
Oct 30 05:15:04.566885 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:15:04.566863    5898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-c3486b-29888955-2n6bs_odh-ai-gateway-infra(d8920d90-c187-4e68-a3a0-ff76a5f28d1b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c3486b-29888955-2n6bs" podUID="d8920d90-c187-4e68-a3a0-ff76a5f28d1b"
Oct 30 05:15:04.568427 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:04.568407    5898 generic.go:356] "Generic (PLEG): container finished" podID="88933a2c-374f-4eb4-98ed-b7f3438774c3" containerID="e0b3a3706d000f663d93a5d42fa6e1ac526c958971287abd328f4d499e664bfb" exitCode=60
Oct 30 05:15:04.568509 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:04.568470    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888955-5njn6" event={"ID":"88933a2c-374f-4eb4-98ed-b7f3438774c3","Type":"ContainerDied","Data":"e0b3a3706d000f663d93a5d42fa6e1ac526c958971287abd328f4d499e664bfb"}
Oct 30 05:15:04.568673 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:04.568658    5898 scope.go:122] "RemoveContainer" containerID="e0b3a3706d000f663d93a5d42fa6e1ac526c958971287abd328f4d499e664bfb"
Oct 30 05:15:04.568935 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:15:04.568882    5898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b21507-29888955-5njn6_odh-ai-gateway-infra(88933a2c-374f-4eb4-98ed-b7f3438774c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888955-5njn6" podUID="88933a2c-374f-4eb4-98ed-b7f3438774c3"
Oct 30 05:15:04.570323 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:04.570301    5898 generic.go:356] "Generic (PLEG): container finished" podID="3ee1e0d5-861e-41ac-929d-cfbdb7d75cd6" containerID="702759b2104dc067a87eb06578211db81a4195213ec7ac7fe61b48514b684d30" exitCode=60
Oct 30 05:15:04.570436 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:04.570384    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ccafe0-29888955-mh52k" event={"ID":"3ee1e0d5-861e-41ac-929d-cfbdb7d75cd6","Type":"ContainerDied","Data":"702759b2104dc067a87eb06578211db81a4195213ec7ac7fe61b48514b684d30"}
Oct 30 05:15:04.570595 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:04.570579    5898 scope.go:122] "RemoveContainer" containerID="702759b2104dc067a87eb06578211db81a4195213ec7ac7fe61b48514b684d30"
Oct 30 05:15:04.570803 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:15:04.570786    5898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-ccafe0-29888955-mh52k_odh-ai-gateway-infra(3ee1e0d5-861e-41ac-929d-cfbdb7d75cd6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ccafe0-29888955-mh52k" podUID="3ee1e0d5-861e-41ac-929d-cfbdb7d75cd6"
Oct 30 05:15:04.581636 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:04.579964    5898 scope.go:122] "RemoveContainer" containerID="25588a28f7f42e424ab29f4aeef8c83e4cb48eab297a249d4aede261bb3e4c1a"
Oct 30 05:15:04.581804 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:04.581781    5898 generic.go:356] "Generic (PLEG): container finished" podID="b453b1b2-fd20-4f49-ba72-9cc2588e4201" containerID="20fde13b30731c3b3fa74f8fb3017441c6e9b518bfd4d8f2c85ec06dc1729f55" exitCode=60
Oct 30 05:15:04.581955 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:04.581925    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888955-5v2p9" event={"ID":"b453b1b2-fd20-4f49-ba72-9cc2588e4201","Type":"ContainerDied","Data":"20fde13b30731c3b3fa74f8fb3017441c6e9b518bfd4d8f2c85ec06dc1729f55"}
Oct 30 05:15:04.582253 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:04.582238    5898 scope.go:122] "RemoveContainer" containerID="20fde13b30731c3b3fa74f8fb3017441c6e9b518bfd4d8f2c85ec06dc1729f55"
Oct 30 05:15:04.582573 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:15:04.582544    5898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888955-5v2p9_odh-ai-gateway-infra(b453b1b2-fd20-4f49-ba72-9cc2588e4201)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888955-5v2p9" podUID="b453b1b2-fd20-4f49-ba72-9cc2588e4201"
Oct 30 05:15:04.584532 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:04.584513    5898 generic.go:356] "Generic (PLEG): container finished" podID="71250a09-9976-4049-9280-de3ccdfd3ab0" containerID="4f36755d68d5c1b9ebeeb62c3498d4731e2efbbc27e8ab5792917dc3520c5702" exitCode=60
Oct 30 05:15:04.584633 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:04.584600    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888955-x2zzg" event={"ID":"71250a09-9976-4049-9280-de3ccdfd3ab0","Type":"ContainerDied","Data":"4f36755d68d5c1b9ebeeb62c3498d4731e2efbbc27e8ab5792917dc3520c5702"}
Oct 30 05:15:04.584848 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:04.584825    5898 scope.go:122] "RemoveContainer" containerID="4f36755d68d5c1b9ebeeb62c3498d4731e2efbbc27e8ab5792917dc3520c5702"
Oct 30 05:15:04.585159 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:15:04.585123    5898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888955-x2zzg_odh-ai-gateway-infra(71250a09-9976-4049-9280-de3ccdfd3ab0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888955-x2zzg" podUID="71250a09-9976-4049-9280-de3ccdfd3ab0"
Oct 30 05:15:04.587271 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:04.587248    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888955-zvsnz" event={"ID":"206ca9bd-dbed-464d-9bff-4a6195a3cda9","Type":"ContainerStarted","Data":"1e77ea3ef432d401c6e1b699081bfcda3757e907a3dfcba61cf0c21b2d3eec1a"}
Oct 30 05:15:04.606711 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:04.606687    5898 scope.go:122] "RemoveContainer" containerID="fcce91d091e8963bccd44af6b473950dcdcc899d17423441c5901acd6df52ed4"
Oct 30 05:15:04.620291 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:04.620270    5898 scope.go:122] "RemoveContainer" containerID="975cb3fbc7eef5408534aeffef65bc831e21cc22d206b3bd887a94808ef38ae4"
Oct 30 05:15:04.633102 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:04.633085    5898 scope.go:122] "RemoveContainer" containerID="007abadfbf113d9cacf7564f129d77f21276ac4ce3dbb8d828331302f4bc2e21"
Oct 30 05:15:04.645790 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:04.645770    5898 scope.go:122] "RemoveContainer" containerID="cd338bc0a223171a46a3ab39b77e4807ff01364766c0c84e646e14fbab280936"
Oct 30 05:15:04.658963 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:04.658943    5898 scope.go:122] "RemoveContainer" containerID="4a100c31dc26c634b99c1d4fe088f10199bf70ede9bb6abe4fd53eb6b692b9b9"
Oct 30 05:15:05.147008 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:05.146973    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:05.595946 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:05.595892    5898 scope.go:122] "RemoveContainer" containerID="e0b3a3706d000f663d93a5d42fa6e1ac526c958971287abd328f4d499e664bfb"
Oct 30 05:15:05.596387 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:15:05.596237    5898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b21507-29888955-5njn6_odh-ai-gateway-infra(88933a2c-374f-4eb4-98ed-b7f3438774c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888955-5njn6" podUID="88933a2c-374f-4eb4-98ed-b7f3438774c3"
Oct 30 05:15:05.597497 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:05.597475    5898 scope.go:122] "RemoveContainer" containerID="702759b2104dc067a87eb06578211db81a4195213ec7ac7fe61b48514b684d30"
Oct 30 05:15:05.597746 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:15:05.597726    5898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-ccafe0-29888955-mh52k_odh-ai-gateway-infra(3ee1e0d5-861e-41ac-929d-cfbdb7d75cd6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ccafe0-29888955-mh52k" podUID="3ee1e0d5-861e-41ac-929d-cfbdb7d75cd6"
Oct 30 05:15:05.599014 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:05.598994    5898 scope.go:122] "RemoveContainer" containerID="20fde13b30731c3b3fa74f8fb3017441c6e9b518bfd4d8f2c85ec06dc1729f55"
Oct 30 05:15:05.599273 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:15:05.599254    5898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888955-5v2p9_odh-ai-gateway-infra(b453b1b2-fd20-4f49-ba72-9cc2588e4201)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888955-5v2p9" podUID="b453b1b2-fd20-4f49-ba72-9cc2588e4201"
Oct 30 05:15:05.600679 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:05.600655    5898 scope.go:122] "RemoveContainer" containerID="4f36755d68d5c1b9ebeeb62c3498d4731e2efbbc27e8ab5792917dc3520c5702"
Oct 30 05:15:05.600933 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:15:05.600890    5898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888955-x2zzg_odh-ai-gateway-infra(71250a09-9976-4049-9280-de3ccdfd3ab0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888955-x2zzg" podUID="71250a09-9976-4049-9280-de3ccdfd3ab0"
Oct 30 05:15:05.602188 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:05.602164    5898 generic.go:356] "Generic (PLEG): container finished" podID="206ca9bd-dbed-464d-9bff-4a6195a3cda9" containerID="1e77ea3ef432d401c6e1b699081bfcda3757e907a3dfcba61cf0c21b2d3eec1a" exitCode=60
Oct 30 05:15:05.602293 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:05.602241    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888955-zvsnz" event={"ID":"206ca9bd-dbed-464d-9bff-4a6195a3cda9","Type":"ContainerDied","Data":"1e77ea3ef432d401c6e1b699081bfcda3757e907a3dfcba61cf0c21b2d3eec1a"}
Oct 30 05:15:05.602359 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:05.602300    5898 scope.go:122] "RemoveContainer" containerID="7a71404924b1cd2a4d42b0610407d0f363b2e4d9acd1566b688c7a0060ffabfc"
Oct 30 05:15:05.602432 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:05.602416    5898 scope.go:122] "RemoveContainer" containerID="1e77ea3ef432d401c6e1b699081bfcda3757e907a3dfcba61cf0c21b2d3eec1a"
Oct 30 05:15:05.602664 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:15:05.602639    5898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-83e83e-29888955-zvsnz_odh-ai-gateway-infra(206ca9bd-dbed-464d-9bff-4a6195a3cda9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888955-zvsnz" podUID="206ca9bd-dbed-464d-9bff-4a6195a3cda9"
Oct 30 05:15:05.604092 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:05.604073    5898 scope.go:122] "RemoveContainer" containerID="583e2cf90c371c32deaee99e660026e3cfc3be763b3cd046faeeb5abccdd3f53"
Oct 30 05:15:05.604298 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:15:05.604281    5898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-36fd3c-29888955-t42l4_odh-ai-gateway-infra(bb3a1f87-17d3-42d5-9019-78a8b6d48825)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-36fd3c-29888955-t42l4" podUID="bb3a1f87-17d3-42d5-9019-78a8b6d48825"
Oct 30 05:15:05.605566 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:05.605547    5898 scope.go:122] "RemoveContainer" containerID="f3592d03c5c21df834d17d39fd6805dd3a2a0c504cdb037d28a3605d207bfbd5"
Oct 30 05:15:05.605771 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:15:05.605749    5898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-010013-29888955-d4m2d_odh-ai-gateway-infra(4bc6a4c7-1a9d-4f6e-bf5b-03138318b1c8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-010013-29888955-d4m2d" podUID="4bc6a4c7-1a9d-4f6e-bf5b-03138318b1c8"
Oct 30 05:15:05.607028 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:05.607014    5898 scope.go:122] "RemoveContainer" containerID="0c60d426f17dda3eedecf4140fc9243f794755a3757b68f9d26079bcc3507023"
Oct 30 05:15:05.607201 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:15:05.607186    5898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-c3486b-29888955-2n6bs_odh-ai-gateway-infra(d8920d90-c187-4e68-a3a0-ff76a5f28d1b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c3486b-29888955-2n6bs" podUID="d8920d90-c187-4e68-a3a0-ff76a5f28d1b"
Oct 30 05:15:06.616457 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:06.616429    5898 scope.go:122] "RemoveContainer" containerID="1e77ea3ef432d401c6e1b699081bfcda3757e907a3dfcba61cf0c21b2d3eec1a"
Oct 30 05:15:06.616984 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:15:06.616775    5898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-83e83e-29888955-zvsnz_odh-ai-gateway-infra(206ca9bd-dbed-464d-9bff-4a6195a3cda9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888955-zvsnz" podUID="206ca9bd-dbed-464d-9bff-4a6195a3cda9"
Oct 30 05:15:06.958093 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:06.957940    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:07.544202 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:07.544166    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:07.673863 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:07.673831    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:07.911354 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:07.911315    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254"]
Oct 30 05:15:07.911679 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:07.911649    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254" podUID="e58b1f9b-6669-485e-93a6-6ef4555b1c8f" containerName="istio-proxy" containerID="cri-o://6df9de15797cfc9dcd85fb3b7070c035553b2c8084fed3b79174407ee0cc6fd4" gracePeriod=30
Oct 30 05:15:07.956161 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:07.956130    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:08.749767 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:08.749739    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:08.870232 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:08.870204    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:09.317801 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:09.317764    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:09.449923 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:09.439689    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:10.945339 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:10.945303    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:12.759330 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:12.759290    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:13.188685 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:13.188658    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254"
Oct 30 05:15:13.261568 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:13.261534    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e58b1f9b-6669-485e-93a6-6ef4555b1c8f-istio-token\" (UniqueName: \"kubernetes.io/projected/e58b1f9b-6669-485e-93a6-6ef4555b1c8f-istio-token\") pod \"e58b1f9b-6669-485e-93a6-6ef4555b1c8f\" (UID: \"e58b1f9b-6669-485e-93a6-6ef4555b1c8f\") "
Oct 30 05:15:13.261725 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:13.261585    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e58b1f9b-6669-485e-93a6-6ef4555b1c8f-kube-api-access-xm7dr\" (UniqueName: \"kubernetes.io/projected/e58b1f9b-6669-485e-93a6-6ef4555b1c8f-kube-api-access-xm7dr\") pod \"e58b1f9b-6669-485e-93a6-6ef4555b1c8f\" (UID: \"e58b1f9b-6669-485e-93a6-6ef4555b1c8f\") "
Oct 30 05:15:13.261725 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:13.261621    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e58b1f9b-6669-485e-93a6-6ef4555b1c8f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58b1f9b-6669-485e-93a6-6ef4555b1c8f-workload-socket\") pod \"e58b1f9b-6669-485e-93a6-6ef4555b1c8f\" (UID: \"e58b1f9b-6669-485e-93a6-6ef4555b1c8f\") "
Oct 30 05:15:13.261725 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:13.261705    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e58b1f9b-6669-485e-93a6-6ef4555b1c8f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e58b1f9b-6669-485e-93a6-6ef4555b1c8f-istio-data\") pod \"e58b1f9b-6669-485e-93a6-6ef4555b1c8f\" (UID: \"e58b1f9b-6669-485e-93a6-6ef4555b1c8f\") "
Oct 30 05:15:13.261930 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:13.261769    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e58b1f9b-6669-485e-93a6-6ef4555b1c8f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58b1f9b-6669-485e-93a6-6ef4555b1c8f-credential-socket\") pod \"e58b1f9b-6669-485e-93a6-6ef4555b1c8f\" (UID: \"e58b1f9b-6669-485e-93a6-6ef4555b1c8f\") "
Oct 30 05:15:13.261930 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:13.261815    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e58b1f9b-6669-485e-93a6-6ef4555b1c8f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e58b1f9b-6669-485e-93a6-6ef4555b1c8f-istio-podinfo\") pod \"e58b1f9b-6669-485e-93a6-6ef4555b1c8f\" (UID: \"e58b1f9b-6669-485e-93a6-6ef4555b1c8f\") "
Oct 30 05:15:13.262120 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:13.262098    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e58b1f9b-6669-485e-93a6-6ef4555b1c8f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e58b1f9b-6669-485e-93a6-6ef4555b1c8f-workload-certs\") pod \"e58b1f9b-6669-485e-93a6-6ef4555b1c8f\" (UID: \"e58b1f9b-6669-485e-93a6-6ef4555b1c8f\") "
Oct 30 05:15:13.262187 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:13.262166    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e58b1f9b-6669-485e-93a6-6ef4555b1c8f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e58b1f9b-6669-485e-93a6-6ef4555b1c8f-istio-envoy\") pod \"e58b1f9b-6669-485e-93a6-6ef4555b1c8f\" (UID: \"e58b1f9b-6669-485e-93a6-6ef4555b1c8f\") "
Oct 30 05:15:13.262241 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:13.262206    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e58b1f9b-6669-485e-93a6-6ef4555b1c8f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e58b1f9b-6669-485e-93a6-6ef4555b1c8f-istiod-ca-cert\") pod \"e58b1f9b-6669-485e-93a6-6ef4555b1c8f\" (UID: \"e58b1f9b-6669-485e-93a6-6ef4555b1c8f\") "
Oct 30 05:15:13.262646 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:13.262374    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e58b1f9b-6669-485e-93a6-6ef4555b1c8f-credential-socket" pod "e58b1f9b-6669-485e-93a6-6ef4555b1c8f" (UID: "e58b1f9b-6669-485e-93a6-6ef4555b1c8f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:15:13.262646 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:13.262433    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e58b1f9b-6669-485e-93a6-6ef4555b1c8f-workload-certs" pod "e58b1f9b-6669-485e-93a6-6ef4555b1c8f" (UID: "e58b1f9b-6669-485e-93a6-6ef4555b1c8f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:15:13.262814 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:13.262652    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e58b1f9b-6669-485e-93a6-6ef4555b1c8f-workload-socket" pod "e58b1f9b-6669-485e-93a6-6ef4555b1c8f" (UID: "e58b1f9b-6669-485e-93a6-6ef4555b1c8f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:15:13.262875 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:13.262824    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e58b1f9b-6669-485e-93a6-6ef4555b1c8f-istiod-ca-cert" pod "e58b1f9b-6669-485e-93a6-6ef4555b1c8f" (UID: "e58b1f9b-6669-485e-93a6-6ef4555b1c8f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:15:13.262966 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:13.262928    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e58b1f9b-6669-485e-93a6-6ef4555b1c8f-istio-data" pod "e58b1f9b-6669-485e-93a6-6ef4555b1c8f" (UID: "e58b1f9b-6669-485e-93a6-6ef4555b1c8f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:15:13.263192 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:13.263171    5898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58b1f9b-6669-485e-93a6-6ef4555b1c8f-workload-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:15:13.263272 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:13.263198    5898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e58b1f9b-6669-485e-93a6-6ef4555b1c8f-istio-data\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:15:13.263272 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:13.263211    5898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58b1f9b-6669-485e-93a6-6ef4555b1c8f-credential-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:15:13.263272 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:13.263225    5898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e58b1f9b-6669-485e-93a6-6ef4555b1c8f-workload-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:15:13.263272 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:13.263239    5898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e58b1f9b-6669-485e-93a6-6ef4555b1c8f-istiod-ca-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:15:13.264461 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:13.264433    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e58b1f9b-6669-485e-93a6-6ef4555b1c8f-kube-api-access-xm7dr" pod "e58b1f9b-6669-485e-93a6-6ef4555b1c8f" (UID: "e58b1f9b-6669-485e-93a6-6ef4555b1c8f"). InnerVolumeSpecName "kube-api-access-xm7dr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:15:13.265344 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:13.265318    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e58b1f9b-6669-485e-93a6-6ef4555b1c8f-istio-podinfo" pod "e58b1f9b-6669-485e-93a6-6ef4555b1c8f" (UID: "e58b1f9b-6669-485e-93a6-6ef4555b1c8f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:15:13.265706 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:13.265680    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e58b1f9b-6669-485e-93a6-6ef4555b1c8f-istio-token" pod "e58b1f9b-6669-485e-93a6-6ef4555b1c8f" (UID: "e58b1f9b-6669-485e-93a6-6ef4555b1c8f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:15:13.265945 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:13.265883    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e58b1f9b-6669-485e-93a6-6ef4555b1c8f-istio-envoy" pod "e58b1f9b-6669-485e-93a6-6ef4555b1c8f" (UID: "e58b1f9b-6669-485e-93a6-6ef4555b1c8f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:15:13.364763 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:13.364735    5898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e58b1f9b-6669-485e-93a6-6ef4555b1c8f-istio-envoy\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:15:13.364763 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:13.364764    5898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e58b1f9b-6669-485e-93a6-6ef4555b1c8f-istio-token\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:15:13.364992 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:13.364773    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm7dr\" (UniqueName: \"kubernetes.io/projected/e58b1f9b-6669-485e-93a6-6ef4555b1c8f-kube-api-access-xm7dr\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:15:13.364992 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:13.364783    5898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e58b1f9b-6669-485e-93a6-6ef4555b1c8f-istio-podinfo\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:15:13.520479 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:13.520448    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:13.620509 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:13.620428    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:13.711466 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:13.711430    5898 generic.go:356] "Generic (PLEG): container finished" podID="e58b1f9b-6669-485e-93a6-6ef4555b1c8f" containerID="6df9de15797cfc9dcd85fb3b7070c035553b2c8084fed3b79174407ee0cc6fd4" exitCode=0
Oct 30 05:15:13.711650 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:13.711493    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254"
Oct 30 05:15:13.711650 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:13.711510    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254" event={"ID":"e58b1f9b-6669-485e-93a6-6ef4555b1c8f","Type":"ContainerDied","Data":"6df9de15797cfc9dcd85fb3b7070c035553b2c8084fed3b79174407ee0cc6fd4"}
Oct 30 05:15:13.711650 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:13.711564    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254" event={"ID":"e58b1f9b-6669-485e-93a6-6ef4555b1c8f","Type":"ContainerDied","Data":"e59ded7f73eae7b0221cffc9b86f5cdbd7e20d5058d5fb6413384e329d68d96e"}
Oct 30 05:15:13.711650 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:13.711600    5898 scope.go:122] "RemoveContainer" containerID="6df9de15797cfc9dcd85fb3b7070c035553b2c8084fed3b79174407ee0cc6fd4"
Oct 30 05:15:13.720774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:13.720744    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:13.725259 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:13.725240    5898 scope.go:122] "RemoveContainer" containerID="9e63a8060c174e9087ce4beb5186c8b62f469a6c3c981c2223382020f6fd7f98"
Oct 30 05:15:13.739782 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:13.739761    5898 scope.go:122] "RemoveContainer" containerID="6df9de15797cfc9dcd85fb3b7070c035553b2c8084fed3b79174407ee0cc6fd4"
Oct 30 05:15:13.740144 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:15:13.740119    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6df9de15797cfc9dcd85fb3b7070c035553b2c8084fed3b79174407ee0cc6fd4\": container with ID starting with 6df9de15797cfc9dcd85fb3b7070c035553b2c8084fed3b79174407ee0cc6fd4 not found: ID does not exist" containerID="6df9de15797cfc9dcd85fb3b7070c035553b2c8084fed3b79174407ee0cc6fd4"
Oct 30 05:15:13.740214 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:13.740159    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6df9de15797cfc9dcd85fb3b7070c035553b2c8084fed3b79174407ee0cc6fd4"} err="failed to get container status \"6df9de15797cfc9dcd85fb3b7070c035553b2c8084fed3b79174407ee0cc6fd4\": rpc error: code = NotFound desc = could not find container \"6df9de15797cfc9dcd85fb3b7070c035553b2c8084fed3b79174407ee0cc6fd4\": container with ID starting with 6df9de15797cfc9dcd85fb3b7070c035553b2c8084fed3b79174407ee0cc6fd4 not found: ID does not exist"
Oct 30 05:15:13.740214 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:13.740186    5898 scope.go:122] "RemoveContainer" containerID="9e63a8060c174e9087ce4beb5186c8b62f469a6c3c981c2223382020f6fd7f98"
Oct 30 05:15:13.740460 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:15:13.740441    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e63a8060c174e9087ce4beb5186c8b62f469a6c3c981c2223382020f6fd7f98\": container with ID starting with 9e63a8060c174e9087ce4beb5186c8b62f469a6c3c981c2223382020f6fd7f98 not found: ID does not exist" containerID="9e63a8060c174e9087ce4beb5186c8b62f469a6c3c981c2223382020f6fd7f98"
Oct 30 05:15:13.740503 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:13.740468    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e63a8060c174e9087ce4beb5186c8b62f469a6c3c981c2223382020f6fd7f98"} err="failed to get container status \"9e63a8060c174e9087ce4beb5186c8b62f469a6c3c981c2223382020f6fd7f98\": rpc error: code = NotFound desc = could not find container \"9e63a8060c174e9087ce4beb5186c8b62f469a6c3c981c2223382020f6fd7f98\": container with ID starting with 9e63a8060c174e9087ce4beb5186c8b62f469a6c3c981c2223382020f6fd7f98 not found: ID does not exist"
Oct 30 05:15:13.776009 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:13.775987    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254"]
Oct 30 05:15:13.784590 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:13.784567    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-4225df-openshift-default-99b8c8fff-7t254"]
Oct 30 05:15:14.848623 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:14.848594    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:14.883620 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:14.883594    5898 scope.go:122] "RemoveContainer" containerID="702759b2104dc067a87eb06578211db81a4195213ec7ac7fe61b48514b684d30"
Oct 30 05:15:14.883975 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:14.883949    5898 scope.go:122] "RemoveContainer" containerID="e0b3a3706d000f663d93a5d42fa6e1ac526c958971287abd328f4d499e664bfb"
Oct 30 05:15:14.893706 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:14.893682    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e58b1f9b-6669-485e-93a6-6ef4555b1c8f" path="/var/lib/kubelet/pods/e58b1f9b-6669-485e-93a6-6ef4555b1c8f/volumes"
Oct 30 05:15:15.451392 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:15.451356    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:15.567947 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:15.567833    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:15.867173 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:15.867134    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888955-5njn6" event={"ID":"88933a2c-374f-4eb4-98ed-b7f3438774c3","Type":"ContainerStarted","Data":"f81fed2469f948f8b5c3792e237bf0f8881a4499a0d54e91a94fe1972c1db3f3"}
Oct 30 05:15:15.883598 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:15.882370    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ccafe0-29888955-mh52k" event={"ID":"3ee1e0d5-861e-41ac-929d-cfbdb7d75cd6","Type":"ContainerStarted","Data":"8cd742f5ec80545bf5a554620f1ba5f0d7eb5da6b1d1097264eedcb013366fea"}
Oct 30 05:15:15.884155 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:15.884095    5898 scope.go:122] "RemoveContainer" containerID="20fde13b30731c3b3fa74f8fb3017441c6e9b518bfd4d8f2c85ec06dc1729f55"
Oct 30 05:15:15.885983 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:15.885259    5898 scope.go:122] "RemoveContainer" containerID="1e77ea3ef432d401c6e1b699081bfcda3757e907a3dfcba61cf0c21b2d3eec1a"
Oct 30 05:15:15.988493 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:15:15.988456    5898 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ee1e0d5_861e_41ac_929d_cfbdb7d75cd6.slice/crio-8cd742f5ec80545bf5a554620f1ba5f0d7eb5da6b1d1097264eedcb013366fea.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ee1e0d5_861e_41ac_929d_cfbdb7d75cd6.slice/crio-conmon-8cd742f5ec80545bf5a554620f1ba5f0d7eb5da6b1d1097264eedcb013366fea.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88933a2c_374f_4eb4_98ed_b7f3438774c3.slice/crio-conmon-f81fed2469f948f8b5c3792e237bf0f8881a4499a0d54e91a94fe1972c1db3f3.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 05:15:16.719027 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:16.718992    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-453511/test-model-453511-kserve-54dcc5c589-wmt2z"]
Oct 30 05:15:16.719648 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:16.719634    5898 cpu_manager.go:414] "containerMap: removing container" podUID="e58b1f9b-6669-485e-93a6-6ef4555b1c8f" containerName="istio-proxy"
Oct 30 05:15:16.719707 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:16.719650    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e58b1f9b-6669-485e-93a6-6ef4555b1c8f" containerName="istio-proxy"
Oct 30 05:15:16.719707 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:16.719671    5898 cpu_manager.go:414] "containerMap: removing container" podUID="e58b1f9b-6669-485e-93a6-6ef4555b1c8f" containerName="istio-proxy"
Oct 30 05:15:16.719707 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:16.719676    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e58b1f9b-6669-485e-93a6-6ef4555b1c8f" containerName="istio-proxy"
Oct 30 05:15:16.719803 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:16.719797    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="e58b1f9b-6669-485e-93a6-6ef4555b1c8f" containerName="istio-proxy"
Oct 30 05:15:16.719834 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:16.719809    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="e58b1f9b-6669-485e-93a6-6ef4555b1c8f" containerName="istio-proxy"
Oct 30 05:15:16.723831 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:16.723806    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-453511/test-model-453511-kserve-54dcc5c589-wmt2z"
Oct 30 05:15:16.743442 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:16.743412    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-453511/test-model-453511-kserve-54dcc5c589-wmt2z"]
Oct 30 05:15:16.748029 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:16.748002    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-453511\"/\"openshift-service-ca.crt\""
Oct 30 05:15:16.748190 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:16.748113    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-453511\"/\"kube-root-ca.crt\""
Oct 30 05:15:16.763945 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:16.755130    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-453511\"/\"test-model-453511-kserve-self-signed-certs\""
Oct 30 05:15:16.763945 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:16.755196    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-453511\"/\"default-dockercfg-8cjlp\""
Oct 30 05:15:16.769246 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:16.769207    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:16.819389 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:16.819355    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e1a48a6-bdd2-46ba-8ddb-15ebca14289a-tls-certs\") pod \"test-model-453511-kserve-54dcc5c589-wmt2z\" (UID: \"0e1a48a6-bdd2-46ba-8ddb-15ebca14289a\") " pod="ai-tenant-e2e-inf-b-453511/test-model-453511-kserve-54dcc5c589-wmt2z"
Oct 30 05:15:16.819549 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:16.819415    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e1a48a6-bdd2-46ba-8ddb-15ebca14289a-home\") pod \"test-model-453511-kserve-54dcc5c589-wmt2z\" (UID: \"0e1a48a6-bdd2-46ba-8ddb-15ebca14289a\") " pod="ai-tenant-e2e-inf-b-453511/test-model-453511-kserve-54dcc5c589-wmt2z"
Oct 30 05:15:16.819657 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:16.819628    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e1a48a6-bdd2-46ba-8ddb-15ebca14289a-tmp-dir\") pod \"test-model-453511-kserve-54dcc5c589-wmt2z\" (UID: \"0e1a48a6-bdd2-46ba-8ddb-15ebca14289a\") " pod="ai-tenant-e2e-inf-b-453511/test-model-453511-kserve-54dcc5c589-wmt2z"
Oct 30 05:15:16.819781 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:16.819760    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e1a48a6-bdd2-46ba-8ddb-15ebca14289a-model-cache\") pod \"test-model-453511-kserve-54dcc5c589-wmt2z\" (UID: \"0e1a48a6-bdd2-46ba-8ddb-15ebca14289a\") " pod="ai-tenant-e2e-inf-b-453511/test-model-453511-kserve-54dcc5c589-wmt2z"
Oct 30 05:15:16.819925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:16.819893    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kzcv\" (UniqueName: \"kubernetes.io/projected/0e1a48a6-bdd2-46ba-8ddb-15ebca14289a-kube-api-access-5kzcv\") pod \"test-model-453511-kserve-54dcc5c589-wmt2z\" (UID: \"0e1a48a6-bdd2-46ba-8ddb-15ebca14289a\") " pod="ai-tenant-e2e-inf-b-453511/test-model-453511-kserve-54dcc5c589-wmt2z"
Oct 30 05:15:16.820066 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:16.820050    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e1a48a6-bdd2-46ba-8ddb-15ebca14289a-dshm\") pod \"test-model-453511-kserve-54dcc5c589-wmt2z\" (UID: \"0e1a48a6-bdd2-46ba-8ddb-15ebca14289a\") " pod="ai-tenant-e2e-inf-b-453511/test-model-453511-kserve-54dcc5c589-wmt2z"
Oct 30 05:15:16.883450 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:16.883423    5898 scope.go:122] "RemoveContainer" containerID="6bdbfa9a81f1a748588262e10295ed98de15546750950da1860f057ca4413329"
Oct 30 05:15:16.908031 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:16.908001    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888955-zvsnz" event={"ID":"206ca9bd-dbed-464d-9bff-4a6195a3cda9","Type":"ContainerStarted","Data":"df668b41ab5ffd67e2ad91a9c1585b1809ec8b1ee5fcf6dc8481fb4c292a30bd"}
Oct 30 05:15:16.909885 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:16.909856    5898 generic.go:356] "Generic (PLEG): container finished" podID="88933a2c-374f-4eb4-98ed-b7f3438774c3" containerID="f81fed2469f948f8b5c3792e237bf0f8881a4499a0d54e91a94fe1972c1db3f3" exitCode=60
Oct 30 05:15:16.910035 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:16.909928    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888955-5njn6" event={"ID":"88933a2c-374f-4eb4-98ed-b7f3438774c3","Type":"ContainerDied","Data":"f81fed2469f948f8b5c3792e237bf0f8881a4499a0d54e91a94fe1972c1db3f3"}
Oct 30 05:15:16.910035 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:16.909988    5898 scope.go:122] "RemoveContainer" containerID="e0b3a3706d000f663d93a5d42fa6e1ac526c958971287abd328f4d499e664bfb"
Oct 30 05:15:16.910402 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:16.910358    5898 scope.go:122] "RemoveContainer" containerID="f81fed2469f948f8b5c3792e237bf0f8881a4499a0d54e91a94fe1972c1db3f3"
Oct 30 05:15:16.910733 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:15:16.910700    5898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b21507-29888955-5njn6_odh-ai-gateway-infra(88933a2c-374f-4eb4-98ed-b7f3438774c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888955-5njn6" podUID="88933a2c-374f-4eb4-98ed-b7f3438774c3"
Oct 30 05:15:16.912316 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:16.912294    5898 generic.go:356] "Generic (PLEG): container finished" podID="3ee1e0d5-861e-41ac-929d-cfbdb7d75cd6" containerID="8cd742f5ec80545bf5a554620f1ba5f0d7eb5da6b1d1097264eedcb013366fea" exitCode=60
Oct 30 05:15:16.912428 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:16.912366    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ccafe0-29888955-mh52k" event={"ID":"3ee1e0d5-861e-41ac-929d-cfbdb7d75cd6","Type":"ContainerDied","Data":"8cd742f5ec80545bf5a554620f1ba5f0d7eb5da6b1d1097264eedcb013366fea"}
Oct 30 05:15:16.912855 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:16.912806    5898 scope.go:122] "RemoveContainer" containerID="8cd742f5ec80545bf5a554620f1ba5f0d7eb5da6b1d1097264eedcb013366fea"
Oct 30 05:15:16.913235 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:15:16.913206    5898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-ccafe0-29888955-mh52k_odh-ai-gateway-infra(3ee1e0d5-861e-41ac-929d-cfbdb7d75cd6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ccafe0-29888955-mh52k" podUID="3ee1e0d5-861e-41ac-929d-cfbdb7d75cd6"
Oct 30 05:15:16.914664 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:16.914642    5898 generic.go:356] "Generic (PLEG): container finished" podID="b453b1b2-fd20-4f49-ba72-9cc2588e4201" containerID="f1b5292272e0a92c7b8da8a10a0d5484adf943e75da7add024d9030da58ec7a0" exitCode=60
Oct 30 05:15:16.914779 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:16.914696    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888955-5v2p9" event={"ID":"b453b1b2-fd20-4f49-ba72-9cc2588e4201","Type":"ContainerDied","Data":"f1b5292272e0a92c7b8da8a10a0d5484adf943e75da7add024d9030da58ec7a0"}
Oct 30 05:15:16.915235 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:16.915217    5898 scope.go:122] "RemoveContainer" containerID="f1b5292272e0a92c7b8da8a10a0d5484adf943e75da7add024d9030da58ec7a0"
Oct 30 05:15:16.915556 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:15:16.915522    5898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888955-5v2p9_odh-ai-gateway-infra(b453b1b2-fd20-4f49-ba72-9cc2588e4201)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888955-5v2p9" podUID="b453b1b2-fd20-4f49-ba72-9cc2588e4201"
Oct 30 05:15:16.921739 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:16.921709    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e1a48a6-bdd2-46ba-8ddb-15ebca14289a-model-cache\") pod \"test-model-453511-kserve-54dcc5c589-wmt2z\" (UID: \"0e1a48a6-bdd2-46ba-8ddb-15ebca14289a\") " pod="ai-tenant-e2e-inf-b-453511/test-model-453511-kserve-54dcc5c589-wmt2z"
Oct 30 05:15:16.921837 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:16.921798    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kzcv\" (UniqueName: \"kubernetes.io/projected/0e1a48a6-bdd2-46ba-8ddb-15ebca14289a-kube-api-access-5kzcv\") pod \"test-model-453511-kserve-54dcc5c589-wmt2z\" (UID: \"0e1a48a6-bdd2-46ba-8ddb-15ebca14289a\") " pod="ai-tenant-e2e-inf-b-453511/test-model-453511-kserve-54dcc5c589-wmt2z"
Oct 30 05:15:16.921976 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:16.921958    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e1a48a6-bdd2-46ba-8ddb-15ebca14289a-dshm\") pod \"test-model-453511-kserve-54dcc5c589-wmt2z\" (UID: \"0e1a48a6-bdd2-46ba-8ddb-15ebca14289a\") " pod="ai-tenant-e2e-inf-b-453511/test-model-453511-kserve-54dcc5c589-wmt2z"
Oct 30 05:15:16.922124 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:16.922102    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e1a48a6-bdd2-46ba-8ddb-15ebca14289a-model-cache\") pod \"test-model-453511-kserve-54dcc5c589-wmt2z\" (UID: \"0e1a48a6-bdd2-46ba-8ddb-15ebca14289a\") " pod="ai-tenant-e2e-inf-b-453511/test-model-453511-kserve-54dcc5c589-wmt2z"
Oct 30 05:15:16.922405 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:16.922384    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e1a48a6-bdd2-46ba-8ddb-15ebca14289a-tls-certs\") pod \"test-model-453511-kserve-54dcc5c589-wmt2z\" (UID: \"0e1a48a6-bdd2-46ba-8ddb-15ebca14289a\") " pod="ai-tenant-e2e-inf-b-453511/test-model-453511-kserve-54dcc5c589-wmt2z"
Oct 30 05:15:16.922851 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:16.922431    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e1a48a6-bdd2-46ba-8ddb-15ebca14289a-home\") pod \"test-model-453511-kserve-54dcc5c589-wmt2z\" (UID: \"0e1a48a6-bdd2-46ba-8ddb-15ebca14289a\") " pod="ai-tenant-e2e-inf-b-453511/test-model-453511-kserve-54dcc5c589-wmt2z"
Oct 30 05:15:16.923120 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:16.923043    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e1a48a6-bdd2-46ba-8ddb-15ebca14289a-home\") pod \"test-model-453511-kserve-54dcc5c589-wmt2z\" (UID: \"0e1a48a6-bdd2-46ba-8ddb-15ebca14289a\") " pod="ai-tenant-e2e-inf-b-453511/test-model-453511-kserve-54dcc5c589-wmt2z"
Oct 30 05:15:16.923309 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:16.923262    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888955-5njn6"]
Oct 30 05:15:16.923777 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:16.923708    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e1a48a6-bdd2-46ba-8ddb-15ebca14289a-tmp-dir\") pod \"test-model-453511-kserve-54dcc5c589-wmt2z\" (UID: \"0e1a48a6-bdd2-46ba-8ddb-15ebca14289a\") " pod="ai-tenant-e2e-inf-b-453511/test-model-453511-kserve-54dcc5c589-wmt2z"
Oct 30 05:15:16.924854 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:16.923912    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e1a48a6-bdd2-46ba-8ddb-15ebca14289a-tmp-dir\") pod \"test-model-453511-kserve-54dcc5c589-wmt2z\" (UID: \"0e1a48a6-bdd2-46ba-8ddb-15ebca14289a\") " pod="ai-tenant-e2e-inf-b-453511/test-model-453511-kserve-54dcc5c589-wmt2z"
Oct 30 05:15:16.924854 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:16.924822    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e1a48a6-bdd2-46ba-8ddb-15ebca14289a-dshm\") pod \"test-model-453511-kserve-54dcc5c589-wmt2z\" (UID: \"0e1a48a6-bdd2-46ba-8ddb-15ebca14289a\") " pod="ai-tenant-e2e-inf-b-453511/test-model-453511-kserve-54dcc5c589-wmt2z"
Oct 30 05:15:16.925290 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:16.925272    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e1a48a6-bdd2-46ba-8ddb-15ebca14289a-tls-certs\") pod \"test-model-453511-kserve-54dcc5c589-wmt2z\" (UID: \"0e1a48a6-bdd2-46ba-8ddb-15ebca14289a\") " pod="ai-tenant-e2e-inf-b-453511/test-model-453511-kserve-54dcc5c589-wmt2z"
Oct 30 05:15:16.942937 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:16.942889    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kzcv\" (UniqueName: \"kubernetes.io/projected/0e1a48a6-bdd2-46ba-8ddb-15ebca14289a-kube-api-access-5kzcv\") pod \"test-model-453511-kserve-54dcc5c589-wmt2z\" (UID: \"0e1a48a6-bdd2-46ba-8ddb-15ebca14289a\") " pod="ai-tenant-e2e-inf-b-453511/test-model-453511-kserve-54dcc5c589-wmt2z"
Oct 30 05:15:16.961996 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:16.961969    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ccafe0-29888955-mh52k"]
Oct 30 05:15:16.979046 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:16.979017    5898 scope.go:122] "RemoveContainer" containerID="702759b2104dc067a87eb06578211db81a4195213ec7ac7fe61b48514b684d30"
Oct 30 05:15:16.980096 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:16.980073    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:17.052927 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:17.049551    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-453511/test-model-453511-kserve-54dcc5c589-wmt2z"
Oct 30 05:15:17.189271 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:17.189112    5898 scope.go:122] "RemoveContainer" containerID="20fde13b30731c3b3fa74f8fb3017441c6e9b518bfd4d8f2c85ec06dc1729f55"
Oct 30 05:15:17.527941 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:17.527885    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-453511/test-model-453511-kserve-54dcc5c589-wmt2z"]
Oct 30 05:15:17.529747 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:15:17.529719    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0e1a48a6_bdd2_46ba_8ddb_15ebca14289a.slice/crio-642647c66343f028e15ad6e35dd2900708ad72d459972957b1183067482cc3a1 WatchSource:0}: Error finding container 642647c66343f028e15ad6e35dd2900708ad72d459972957b1183067482cc3a1: Status 404 returned error can't find the container with id 642647c66343f028e15ad6e35dd2900708ad72d459972957b1183067482cc3a1
Oct 30 05:15:17.882678 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:17.882653    5898 scope.go:122] "RemoveContainer" containerID="583e2cf90c371c32deaee99e660026e3cfc3be763b3cd046faeeb5abccdd3f53"
Oct 30 05:15:17.882816 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:17.882785    5898 scope.go:122] "RemoveContainer" containerID="f3592d03c5c21df834d17d39fd6805dd3a2a0c504cdb037d28a3605d207bfbd5"
Oct 30 05:15:17.882913 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:17.882840    5898 scope.go:122] "RemoveContainer" containerID="4f36755d68d5c1b9ebeeb62c3498d4731e2efbbc27e8ab5792917dc3520c5702"
Oct 30 05:15:17.883078 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:17.883061    5898 scope.go:122] "RemoveContainer" containerID="0c60d426f17dda3eedecf4140fc9243f794755a3757b68f9d26079bcc3507023"
Oct 30 05:15:17.927271 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:17.927245    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-453511/test-model-453511-kserve-54dcc5c589-wmt2z" event={"ID":"0e1a48a6-bdd2-46ba-8ddb-15ebca14289a","Type":"ContainerStarted","Data":"bcb648d88b38cde0b2d7e5fb22aa9b1844b6ecd64bca48ca4cd61903428568c3"}
Oct 30 05:15:17.927559 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:17.927291    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-453511/test-model-453511-kserve-54dcc5c589-wmt2z" event={"ID":"0e1a48a6-bdd2-46ba-8ddb-15ebca14289a","Type":"ContainerStarted","Data":"642647c66343f028e15ad6e35dd2900708ad72d459972957b1183067482cc3a1"}
Oct 30 05:15:17.938029 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:17.937214    5898 generic.go:356] "Generic (PLEG): container finished" podID="206ca9bd-dbed-464d-9bff-4a6195a3cda9" containerID="df668b41ab5ffd67e2ad91a9c1585b1809ec8b1ee5fcf6dc8481fb4c292a30bd" exitCode=60
Oct 30 05:15:17.938029 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:17.937304    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888955-zvsnz" event={"ID":"206ca9bd-dbed-464d-9bff-4a6195a3cda9","Type":"ContainerDied","Data":"df668b41ab5ffd67e2ad91a9c1585b1809ec8b1ee5fcf6dc8481fb4c292a30bd"}
Oct 30 05:15:17.938029 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:17.937352    5898 scope.go:122] "RemoveContainer" containerID="1e77ea3ef432d401c6e1b699081bfcda3757e907a3dfcba61cf0c21b2d3eec1a"
Oct 30 05:15:17.938029 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:17.937766    5898 scope.go:122] "RemoveContainer" containerID="df668b41ab5ffd67e2ad91a9c1585b1809ec8b1ee5fcf6dc8481fb4c292a30bd"
Oct 30 05:15:17.938294 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:15:17.938169    5898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-83e83e-29888955-zvsnz_odh-ai-gateway-infra(206ca9bd-dbed-464d-9bff-4a6195a3cda9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888955-zvsnz" podUID="206ca9bd-dbed-464d-9bff-4a6195a3cda9"
Oct 30 05:15:17.941716 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:17.941681    5898 generic.go:356] "Generic (PLEG): container finished" podID="938a8472-fb50-452a-bef7-416b671f3362" containerID="d87239578c586227c1c6edbd5ff676a1cb66efc91a740bb7ce1d66544c9719aa" exitCode=60
Oct 30 05:15:17.941836 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:17.941748    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888955-gw8mj" event={"ID":"938a8472-fb50-452a-bef7-416b671f3362","Type":"ContainerDied","Data":"d87239578c586227c1c6edbd5ff676a1cb66efc91a740bb7ce1d66544c9719aa"}
Oct 30 05:15:17.942252 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:17.942231    5898 scope.go:122] "RemoveContainer" containerID="d87239578c586227c1c6edbd5ff676a1cb66efc91a740bb7ce1d66544c9719aa"
Oct 30 05:15:17.942638 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:15:17.942614    5898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888955-gw8mj_odh-ai-gateway-infra(938a8472-fb50-452a-bef7-416b671f3362)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888955-gw8mj" podUID="938a8472-fb50-452a-bef7-416b671f3362"
Oct 30 05:15:17.962182 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:17.959934    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888955-zvsnz"]
Oct 30 05:15:17.966144 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:17.966090    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-453511/test-model-453511-kserve-54dcc5c589-wmt2z" podStartSLOduration=1.9660720299999999 podStartE2EDuration="1.96607203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:15:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:15:17.965936062 +0000 UTC m=+1847.189755963" watchObservedRunningTime="2026-10-30 05:15:17.96607203 +0000 UTC m=+1847.189891932"
Oct 30 05:15:17.986086 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:17.985674    5898 scope.go:122] "RemoveContainer" containerID="6bdbfa9a81f1a748588262e10295ed98de15546750950da1860f057ca4413329"
Oct 30 05:15:17.997448 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:17.997423    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888955-5v2p9"]
Oct 30 05:15:18.051330 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:18.051295    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-453511/test-model-453511-kserve-54dcc5c589-wmt2z"
Oct 30 05:15:18.270110 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:18.270086    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ccafe0-29888955-mh52k"
Oct 30 05:15:18.278852 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:18.278768    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888955-5njn6"
Oct 30 05:15:18.343981 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:18.343953    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888955-5v2p9"
Oct 30 05:15:18.393640 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:18.392895    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ee1e0d5-861e-41ac-929d-cfbdb7d75cd6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ee1e0d5-861e-41ac-929d-cfbdb7d75cd6-maas-api-service-ca\") pod \"3ee1e0d5-861e-41ac-929d-cfbdb7d75cd6\" (UID: \"3ee1e0d5-861e-41ac-929d-cfbdb7d75cd6\") "
Oct 30 05:15:18.393640 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:18.393053    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b453b1b2-fd20-4f49-ba72-9cc2588e4201-kube-api-access-297z9\" (UniqueName: \"kubernetes.io/projected/b453b1b2-fd20-4f49-ba72-9cc2588e4201-kube-api-access-297z9\") pod \"b453b1b2-fd20-4f49-ba72-9cc2588e4201\" (UID: \"b453b1b2-fd20-4f49-ba72-9cc2588e4201\") "
Oct 30 05:15:18.393640 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:18.393108    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88933a2c-374f-4eb4-98ed-b7f3438774c3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88933a2c-374f-4eb4-98ed-b7f3438774c3-maas-api-service-ca\") pod \"88933a2c-374f-4eb4-98ed-b7f3438774c3\" (UID: \"88933a2c-374f-4eb4-98ed-b7f3438774c3\") "
Oct 30 05:15:18.393640 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:18.393140    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88933a2c-374f-4eb4-98ed-b7f3438774c3-kube-api-access-w8pk9\" (UniqueName: \"kubernetes.io/projected/88933a2c-374f-4eb4-98ed-b7f3438774c3-kube-api-access-w8pk9\") pod \"88933a2c-374f-4eb4-98ed-b7f3438774c3\" (UID: \"88933a2c-374f-4eb4-98ed-b7f3438774c3\") "
Oct 30 05:15:18.393640 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:18.393280    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b453b1b2-fd20-4f49-ba72-9cc2588e4201-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b453b1b2-fd20-4f49-ba72-9cc2588e4201-maas-api-service-ca\") pod \"b453b1b2-fd20-4f49-ba72-9cc2588e4201\" (UID: \"b453b1b2-fd20-4f49-ba72-9cc2588e4201\") "
Oct 30 05:15:18.393640 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:18.393370    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ee1e0d5-861e-41ac-929d-cfbdb7d75cd6-kube-api-access-5fh6d\" (UniqueName: \"kubernetes.io/projected/3ee1e0d5-861e-41ac-929d-cfbdb7d75cd6-kube-api-access-5fh6d\") pod \"3ee1e0d5-861e-41ac-929d-cfbdb7d75cd6\" (UID: \"3ee1e0d5-861e-41ac-929d-cfbdb7d75cd6\") "
Oct 30 05:15:18.393640 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:18.393432    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ee1e0d5-861e-41ac-929d-cfbdb7d75cd6-maas-api-service-ca" pod "3ee1e0d5-861e-41ac-929d-cfbdb7d75cd6" (UID: "3ee1e0d5-861e-41ac-929d-cfbdb7d75cd6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:15:18.394135 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:18.394091    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88933a2c-374f-4eb4-98ed-b7f3438774c3-maas-api-service-ca" pod "88933a2c-374f-4eb4-98ed-b7f3438774c3" (UID: "88933a2c-374f-4eb4-98ed-b7f3438774c3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:15:18.394229 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:18.394209    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ee1e0d5-861e-41ac-929d-cfbdb7d75cd6-maas-api-service-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:15:18.394308 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:18.394233    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88933a2c-374f-4eb4-98ed-b7f3438774c3-maas-api-service-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:15:18.394657 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:18.394634    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b453b1b2-fd20-4f49-ba72-9cc2588e4201-maas-api-service-ca" pod "b453b1b2-fd20-4f49-ba72-9cc2588e4201" (UID: "b453b1b2-fd20-4f49-ba72-9cc2588e4201"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:15:18.397828 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:18.397772    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b453b1b2-fd20-4f49-ba72-9cc2588e4201-kube-api-access-297z9" pod "b453b1b2-fd20-4f49-ba72-9cc2588e4201" (UID: "b453b1b2-fd20-4f49-ba72-9cc2588e4201"). InnerVolumeSpecName "kube-api-access-297z9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:15:18.398299 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:18.398277    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ee1e0d5-861e-41ac-929d-cfbdb7d75cd6-kube-api-access-5fh6d" pod "3ee1e0d5-861e-41ac-929d-cfbdb7d75cd6" (UID: "3ee1e0d5-861e-41ac-929d-cfbdb7d75cd6"). InnerVolumeSpecName "kube-api-access-5fh6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:15:18.405185 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:18.405145    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88933a2c-374f-4eb4-98ed-b7f3438774c3-kube-api-access-w8pk9" pod "88933a2c-374f-4eb4-98ed-b7f3438774c3" (UID: "88933a2c-374f-4eb4-98ed-b7f3438774c3"). InnerVolumeSpecName "kube-api-access-w8pk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:15:18.496084 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:18.496060    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-297z9\" (UniqueName: \"kubernetes.io/projected/b453b1b2-fd20-4f49-ba72-9cc2588e4201-kube-api-access-297z9\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:15:18.496084 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:18.496085    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8pk9\" (UniqueName: \"kubernetes.io/projected/88933a2c-374f-4eb4-98ed-b7f3438774c3-kube-api-access-w8pk9\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:15:18.496215 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:18.496094    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b453b1b2-fd20-4f49-ba72-9cc2588e4201-maas-api-service-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:15:18.496215 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:18.496104    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fh6d\" (UniqueName: \"kubernetes.io/projected/3ee1e0d5-861e-41ac-929d-cfbdb7d75cd6-kube-api-access-5fh6d\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:15:18.951776 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:18.951746    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888955-5v2p9"
Oct 30 05:15:18.952230 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:18.951770    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888955-5v2p9" event={"ID":"b453b1b2-fd20-4f49-ba72-9cc2588e4201","Type":"ContainerDied","Data":"acf7131df3e1288c21d8835a26cdce8c86ef6587a99379cb304c5d09e70279fd"}
Oct 30 05:15:18.952230 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:18.951836    5898 scope.go:122] "RemoveContainer" containerID="f1b5292272e0a92c7b8da8a10a0d5484adf943e75da7add024d9030da58ec7a0"
Oct 30 05:15:18.954184 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:18.954162    5898 generic.go:356] "Generic (PLEG): container finished" podID="71250a09-9976-4049-9280-de3ccdfd3ab0" containerID="8255b107b512434bc3f0b9e7dd6b815a611727f3bc0b9da3829c8ca321d82715" exitCode=60
Oct 30 05:15:18.954291 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:18.954231    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888955-x2zzg" event={"ID":"71250a09-9976-4049-9280-de3ccdfd3ab0","Type":"ContainerDied","Data":"8255b107b512434bc3f0b9e7dd6b815a611727f3bc0b9da3829c8ca321d82715"}
Oct 30 05:15:18.954708 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:18.954674    5898 scope.go:122] "RemoveContainer" containerID="8255b107b512434bc3f0b9e7dd6b815a611727f3bc0b9da3829c8ca321d82715"
Oct 30 05:15:18.955127 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:15:18.955030    5898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888955-x2zzg_odh-ai-gateway-infra(71250a09-9976-4049-9280-de3ccdfd3ab0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888955-x2zzg" podUID="71250a09-9976-4049-9280-de3ccdfd3ab0"
Oct 30 05:15:18.960396 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:18.960375    5898 generic.go:356] "Generic (PLEG): container finished" podID="bb3a1f87-17d3-42d5-9019-78a8b6d48825" containerID="1bbad2d94e4e097f27d2585c45774e2688a2183fe6167fda9a1069b44c1e8b0b" exitCode=60
Oct 30 05:15:18.960520 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:18.960443    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-36fd3c-29888955-t42l4" event={"ID":"bb3a1f87-17d3-42d5-9019-78a8b6d48825","Type":"ContainerDied","Data":"1bbad2d94e4e097f27d2585c45774e2688a2183fe6167fda9a1069b44c1e8b0b"}
Oct 30 05:15:18.960927 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:18.960886    5898 scope.go:122] "RemoveContainer" containerID="1bbad2d94e4e097f27d2585c45774e2688a2183fe6167fda9a1069b44c1e8b0b"
Oct 30 05:15:18.961239 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:15:18.961213    5898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-36fd3c-29888955-t42l4_odh-ai-gateway-infra(bb3a1f87-17d3-42d5-9019-78a8b6d48825)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-36fd3c-29888955-t42l4" podUID="bb3a1f87-17d3-42d5-9019-78a8b6d48825"
Oct 30 05:15:18.962415 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:18.962392    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-010013-29888955-d4m2d" event={"ID":"4bc6a4c7-1a9d-4f6e-bf5b-03138318b1c8","Type":"ContainerStarted","Data":"ed2ab0627f24cd88c5e77a72214cc93a4d1ddfb8062fd185e637644a0eeb2aac"}
Oct 30 05:15:18.964451 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:18.964420    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c3486b-29888955-2n6bs" event={"ID":"d8920d90-c187-4e68-a3a0-ff76a5f28d1b","Type":"ContainerStarted","Data":"9da1d96e863b17f115fad4a152f522ca9365cea8f7c3b6c9beee8f6c6d154ff8"}
Oct 30 05:15:18.966188 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:18.966168    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888955-5njn6" event={"ID":"88933a2c-374f-4eb4-98ed-b7f3438774c3","Type":"ContainerDied","Data":"8f41f38571d67e130334c9799b10caff28096a06824e05486c6e070f61347020"}
Oct 30 05:15:18.966331 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:18.966174    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888955-5njn6"
Oct 30 05:15:18.967844 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:18.967827    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ccafe0-29888955-mh52k"
Oct 30 05:15:18.968168 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:18.968082    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ccafe0-29888955-mh52k" event={"ID":"3ee1e0d5-861e-41ac-929d-cfbdb7d75cd6","Type":"ContainerDied","Data":"ee2b85897072cdc3afd46644396bc90458389d9e4ce96df50922f62a7e2dc9fe"}
Oct 30 05:15:19.026783 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:19.026752    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:15:19.060142 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:19.060102    5898 scope.go:122] "RemoveContainer" containerID="4f36755d68d5c1b9ebeeb62c3498d4731e2efbbc27e8ab5792917dc3520c5702"
Oct 30 05:15:19.061502 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:19.061290    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888955-gw8mj"]
Oct 30 05:15:19.102836 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:19.101548    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:19.164955 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:19.161061    5898 scope.go:122] "RemoveContainer" containerID="583e2cf90c371c32deaee99e660026e3cfc3be763b3cd046faeeb5abccdd3f53"
Oct 30 05:15:19.304171 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:19.304132    5898 scope.go:122] "RemoveContainer" containerID="f81fed2469f948f8b5c3792e237bf0f8881a4499a0d54e91a94fe1972c1db3f3"
Oct 30 05:15:19.328809 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:19.328013    5898 scope.go:122] "RemoveContainer" containerID="8cd742f5ec80545bf5a554620f1ba5f0d7eb5da6b1d1097264eedcb013366fea"
Oct 30 05:15:19.386239 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:19.386209    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888955-5v2p9"]
Oct 30 05:15:19.407712 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:19.407679    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-93bad9-29888955-5v2p9"]
Oct 30 05:15:19.420991 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:19.420969    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888955-zvsnz"
Oct 30 05:15:19.434229 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:19.434195    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888955-gw8mj"
Oct 30 05:15:19.438979 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:19.438949    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888955-5njn6"]
Oct 30 05:15:19.448858 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:19.448834    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888955-5njn6"]
Oct 30 05:15:19.484357 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:19.484335    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ccafe0-29888955-mh52k"]
Oct 30 05:15:19.487744 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:19.487721    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ccafe0-29888955-mh52k"]
Oct 30 05:15:19.518423 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:19.518391    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/938a8472-fb50-452a-bef7-416b671f3362-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/938a8472-fb50-452a-bef7-416b671f3362-maas-api-service-ca\") pod \"938a8472-fb50-452a-bef7-416b671f3362\" (UID: \"938a8472-fb50-452a-bef7-416b671f3362\") "
Oct 30 05:15:19.518541 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:19.518512    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/938a8472-fb50-452a-bef7-416b671f3362-kube-api-access-w5hld\" (UniqueName: \"kubernetes.io/projected/938a8472-fb50-452a-bef7-416b671f3362-kube-api-access-w5hld\") pod \"938a8472-fb50-452a-bef7-416b671f3362\" (UID: \"938a8472-fb50-452a-bef7-416b671f3362\") "
Oct 30 05:15:19.518674 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:19.518660    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/206ca9bd-dbed-464d-9bff-4a6195a3cda9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/206ca9bd-dbed-464d-9bff-4a6195a3cda9-maas-api-service-ca\") pod \"206ca9bd-dbed-464d-9bff-4a6195a3cda9\" (UID: \"206ca9bd-dbed-464d-9bff-4a6195a3cda9\") "
Oct 30 05:15:19.518789 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:19.518762    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/206ca9bd-dbed-464d-9bff-4a6195a3cda9-kube-api-access-lx7rv\" (UniqueName: \"kubernetes.io/projected/206ca9bd-dbed-464d-9bff-4a6195a3cda9-kube-api-access-lx7rv\") pod \"206ca9bd-dbed-464d-9bff-4a6195a3cda9\" (UID: \"206ca9bd-dbed-464d-9bff-4a6195a3cda9\") "
Oct 30 05:15:19.518789 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:19.518771    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/938a8472-fb50-452a-bef7-416b671f3362-maas-api-service-ca" pod "938a8472-fb50-452a-bef7-416b671f3362" (UID: "938a8472-fb50-452a-bef7-416b671f3362"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:15:19.519086 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:19.519063    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/206ca9bd-dbed-464d-9bff-4a6195a3cda9-maas-api-service-ca" pod "206ca9bd-dbed-464d-9bff-4a6195a3cda9" (UID: "206ca9bd-dbed-464d-9bff-4a6195a3cda9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:15:19.519447 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:19.519422    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/938a8472-fb50-452a-bef7-416b671f3362-maas-api-service-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:15:19.519582 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:19.519450    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/206ca9bd-dbed-464d-9bff-4a6195a3cda9-maas-api-service-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:15:19.521227 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:19.521202    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/938a8472-fb50-452a-bef7-416b671f3362-kube-api-access-w5hld" pod "938a8472-fb50-452a-bef7-416b671f3362" (UID: "938a8472-fb50-452a-bef7-416b671f3362"). InnerVolumeSpecName "kube-api-access-w5hld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:15:19.521320 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:19.521238    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/206ca9bd-dbed-464d-9bff-4a6195a3cda9-kube-api-access-lx7rv" pod "206ca9bd-dbed-464d-9bff-4a6195a3cda9" (UID: "206ca9bd-dbed-464d-9bff-4a6195a3cda9"). InnerVolumeSpecName "kube-api-access-lx7rv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:15:19.620406 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:19.620371    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx7rv\" (UniqueName: \"kubernetes.io/projected/206ca9bd-dbed-464d-9bff-4a6195a3cda9-kube-api-access-lx7rv\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:15:19.620406 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:19.620400    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5hld\" (UniqueName: \"kubernetes.io/projected/938a8472-fb50-452a-bef7-416b671f3362-kube-api-access-w5hld\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:15:19.713025 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:19.712935    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:20.010730 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:20.010694    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888955-zvsnz" event={"ID":"206ca9bd-dbed-464d-9bff-4a6195a3cda9","Type":"ContainerDied","Data":"788d8b31b96384a728fb4b5f0ca5411675128561f9247fe1b215e50e0a50eba9"}
Oct 30 05:15:20.011410 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:20.011388    5898 scope.go:122] "RemoveContainer" containerID="df668b41ab5ffd67e2ad91a9c1585b1809ec8b1ee5fcf6dc8481fb4c292a30bd"
Oct 30 05:15:20.011690 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:20.011661    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888955-zvsnz"
Oct 30 05:15:20.030067 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:20.029352    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888955-gw8mj" event={"ID":"938a8472-fb50-452a-bef7-416b671f3362","Type":"ContainerDied","Data":"394342e6e1c2f5ed539d8780c928fcbd4dfc70720e33cb488d4b12afe532b4bf"}
Oct 30 05:15:20.030067 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:20.029476    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888955-gw8mj"
Oct 30 05:15:20.042238 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:20.040501    5898 generic.go:356] "Generic (PLEG): container finished" podID="4bc6a4c7-1a9d-4f6e-bf5b-03138318b1c8" containerID="ed2ab0627f24cd88c5e77a72214cc93a4d1ddfb8062fd185e637644a0eeb2aac" exitCode=60
Oct 30 05:15:20.042238 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:20.040592    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-010013-29888955-d4m2d" event={"ID":"4bc6a4c7-1a9d-4f6e-bf5b-03138318b1c8","Type":"ContainerDied","Data":"ed2ab0627f24cd88c5e77a72214cc93a4d1ddfb8062fd185e637644a0eeb2aac"}
Oct 30 05:15:20.042238 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:20.041111    5898 scope.go:122] "RemoveContainer" containerID="ed2ab0627f24cd88c5e77a72214cc93a4d1ddfb8062fd185e637644a0eeb2aac"
Oct 30 05:15:20.042238 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:15:20.041578    5898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-010013-29888955-d4m2d_odh-ai-gateway-infra(4bc6a4c7-1a9d-4f6e-bf5b-03138318b1c8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-010013-29888955-d4m2d" podUID="4bc6a4c7-1a9d-4f6e-bf5b-03138318b1c8"
Oct 30 05:15:20.044563 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:20.044537    5898 generic.go:356] "Generic (PLEG): container finished" podID="d8920d90-c187-4e68-a3a0-ff76a5f28d1b" containerID="9da1d96e863b17f115fad4a152f522ca9365cea8f7c3b6c9beee8f6c6d154ff8" exitCode=60
Oct 30 05:15:20.044998 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:20.044970    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c3486b-29888955-2n6bs" event={"ID":"d8920d90-c187-4e68-a3a0-ff76a5f28d1b","Type":"ContainerDied","Data":"9da1d96e863b17f115fad4a152f522ca9365cea8f7c3b6c9beee8f6c6d154ff8"}
Oct 30 05:15:20.045458 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:20.045396    5898 scope.go:122] "RemoveContainer" containerID="9da1d96e863b17f115fad4a152f522ca9365cea8f7c3b6c9beee8f6c6d154ff8"
Oct 30 05:15:20.045760 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:15:20.045723    5898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-c3486b-29888955-2n6bs_odh-ai-gateway-infra(d8920d90-c187-4e68-a3a0-ff76a5f28d1b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c3486b-29888955-2n6bs" podUID="d8920d90-c187-4e68-a3a0-ff76a5f28d1b"
Oct 30 05:15:20.057405 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:20.056762    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c3486b-29888955-2n6bs"]
Oct 30 05:15:20.059486 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:20.059464    5898 scope.go:122] "RemoveContainer" containerID="d87239578c586227c1c6edbd5ff676a1cb66efc91a740bb7ce1d66544c9719aa"
Oct 30 05:15:20.111070 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:20.111035    5898 scope.go:122] "RemoveContainer" containerID="f3592d03c5c21df834d17d39fd6805dd3a2a0c504cdb037d28a3605d207bfbd5"
Oct 30 05:15:20.146757 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:20.146721    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-010013-29888955-d4m2d"]
Oct 30 05:15:20.172108 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:20.171688    5898 scope.go:122] "RemoveContainer" containerID="0c60d426f17dda3eedecf4140fc9243f794755a3757b68f9d26079bcc3507023"
Oct 30 05:15:20.213039 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:20.212970    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888955-zvsnz"]
Oct 30 05:15:20.315558 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:20.315520    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888955-zvsnz"]
Oct 30 05:15:20.324945 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:20.324867    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888955-gw8mj"]
Oct 30 05:15:20.330792 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:20.330761    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888955-gw8mj"]
Oct 30 05:15:20.333958 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:20.333931    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888955-x2zzg"]
Oct 30 05:15:20.343832 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:20.343806    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-36fd3c-29888955-t42l4"]
Oct 30 05:15:20.547071 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:20.547038    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888955-x2zzg"
Oct 30 05:15:20.593498 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:20.593042    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-36fd3c-29888955-t42l4"
Oct 30 05:15:20.730961 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:20.724521    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71250a09-9976-4049-9280-de3ccdfd3ab0-kube-api-access-lbtv4\" (UniqueName: \"kubernetes.io/projected/71250a09-9976-4049-9280-de3ccdfd3ab0-kube-api-access-lbtv4\") pod \"71250a09-9976-4049-9280-de3ccdfd3ab0\" (UID: \"71250a09-9976-4049-9280-de3ccdfd3ab0\") "
Oct 30 05:15:20.730961 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:20.724585    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb3a1f87-17d3-42d5-9019-78a8b6d48825-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb3a1f87-17d3-42d5-9019-78a8b6d48825-maas-api-service-ca\") pod \"bb3a1f87-17d3-42d5-9019-78a8b6d48825\" (UID: \"bb3a1f87-17d3-42d5-9019-78a8b6d48825\") "
Oct 30 05:15:20.730961 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:20.724664    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb3a1f87-17d3-42d5-9019-78a8b6d48825-kube-api-access-g77m9\" (UniqueName: \"kubernetes.io/projected/bb3a1f87-17d3-42d5-9019-78a8b6d48825-kube-api-access-g77m9\") pod \"bb3a1f87-17d3-42d5-9019-78a8b6d48825\" (UID: \"bb3a1f87-17d3-42d5-9019-78a8b6d48825\") "
Oct 30 05:15:20.730961 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:20.724750    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71250a09-9976-4049-9280-de3ccdfd3ab0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71250a09-9976-4049-9280-de3ccdfd3ab0-maas-api-service-ca\") pod \"71250a09-9976-4049-9280-de3ccdfd3ab0\" (UID: \"71250a09-9976-4049-9280-de3ccdfd3ab0\") "
Oct 30 05:15:20.730961 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:20.726615    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71250a09-9976-4049-9280-de3ccdfd3ab0-maas-api-service-ca" pod "71250a09-9976-4049-9280-de3ccdfd3ab0" (UID: "71250a09-9976-4049-9280-de3ccdfd3ab0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:15:20.739341 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:20.739298    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb3a1f87-17d3-42d5-9019-78a8b6d48825-kube-api-access-g77m9" pod "bb3a1f87-17d3-42d5-9019-78a8b6d48825" (UID: "bb3a1f87-17d3-42d5-9019-78a8b6d48825"). InnerVolumeSpecName "kube-api-access-g77m9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:15:20.741492 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:20.741466    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71250a09-9976-4049-9280-de3ccdfd3ab0-kube-api-access-lbtv4" pod "71250a09-9976-4049-9280-de3ccdfd3ab0" (UID: "71250a09-9976-4049-9280-de3ccdfd3ab0"). InnerVolumeSpecName "kube-api-access-lbtv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:15:20.798972 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:20.798324    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb3a1f87-17d3-42d5-9019-78a8b6d48825-maas-api-service-ca" pod "bb3a1f87-17d3-42d5-9019-78a8b6d48825" (UID: "bb3a1f87-17d3-42d5-9019-78a8b6d48825"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:15:20.827064 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:20.827028    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbtv4\" (UniqueName: \"kubernetes.io/projected/71250a09-9976-4049-9280-de3ccdfd3ab0-kube-api-access-lbtv4\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:15:20.827064 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:20.827062    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb3a1f87-17d3-42d5-9019-78a8b6d48825-maas-api-service-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:15:20.827246 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:20.827078    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g77m9\" (UniqueName: \"kubernetes.io/projected/bb3a1f87-17d3-42d5-9019-78a8b6d48825-kube-api-access-g77m9\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:15:20.827246 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:20.827092    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71250a09-9976-4049-9280-de3ccdfd3ab0-maas-api-service-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:15:20.904673 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:20.904640    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="206ca9bd-dbed-464d-9bff-4a6195a3cda9" path="/var/lib/kubelet/pods/206ca9bd-dbed-464d-9bff-4a6195a3cda9/volumes"
Oct 30 05:15:20.905164 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:20.905147    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ee1e0d5-861e-41ac-929d-cfbdb7d75cd6" path="/var/lib/kubelet/pods/3ee1e0d5-861e-41ac-929d-cfbdb7d75cd6/volumes"
Oct 30 05:15:20.905560 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:20.905546    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88933a2c-374f-4eb4-98ed-b7f3438774c3" path="/var/lib/kubelet/pods/88933a2c-374f-4eb4-98ed-b7f3438774c3/volumes"
Oct 30 05:15:20.906326 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:20.906311    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="938a8472-fb50-452a-bef7-416b671f3362" path="/var/lib/kubelet/pods/938a8472-fb50-452a-bef7-416b671f3362/volumes"
Oct 30 05:15:20.906695 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:20.906683    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b453b1b2-fd20-4f49-ba72-9cc2588e4201" path="/var/lib/kubelet/pods/b453b1b2-fd20-4f49-ba72-9cc2588e4201/volumes"
Oct 30 05:15:21.058611 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:21.058533    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-36fd3c-29888955-t42l4"
Oct 30 05:15:21.059126 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:21.058526    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-36fd3c-29888955-t42l4" event={"ID":"bb3a1f87-17d3-42d5-9019-78a8b6d48825","Type":"ContainerDied","Data":"c2cf763ca81344a8e61ef7d5240614716c3de14eae39242a49546b4ecc8eee62"}
Oct 30 05:15:21.059126 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:21.058675    5898 scope.go:122] "RemoveContainer" containerID="1bbad2d94e4e097f27d2585c45774e2688a2183fe6167fda9a1069b44c1e8b0b"
Oct 30 05:15:21.065303 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:21.065282    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888955-x2zzg" event={"ID":"71250a09-9976-4049-9280-de3ccdfd3ab0","Type":"ContainerDied","Data":"bd56d68a831e4947e2a464dd41dec09d59129d96e1fd7af581a47fb5c97b74cd"}
Oct 30 05:15:21.065417 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:21.065327    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888955-x2zzg"
Oct 30 05:15:21.070879 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:21.070825    5898 scope.go:122] "RemoveContainer" containerID="8255b107b512434bc3f0b9e7dd6b815a611727f3bc0b9da3829c8ca321d82715"
Oct 30 05:15:21.109560 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:21.109517    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-36fd3c-29888955-t42l4"]
Oct 30 05:15:21.119963 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:21.119929    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-36fd3c-29888955-t42l4"]
Oct 30 05:15:21.251948 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:21.251401    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888955-x2zzg"]
Oct 30 05:15:21.255637 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:21.255611    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888955-x2zzg"]
Oct 30 05:15:21.259350 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:21.259328    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-010013-29888955-d4m2d"
Oct 30 05:15:21.287327 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:21.287304    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c3486b-29888955-2n6bs"
Oct 30 05:15:21.294076 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:21.294041    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:21.438347 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:21.438316    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8920d90-c187-4e68-a3a0-ff76a5f28d1b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8920d90-c187-4e68-a3a0-ff76a5f28d1b-maas-api-service-ca\") pod \"d8920d90-c187-4e68-a3a0-ff76a5f28d1b\" (UID: \"d8920d90-c187-4e68-a3a0-ff76a5f28d1b\") "
Oct 30 05:15:21.438521 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:21.438450    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bc6a4c7-1a9d-4f6e-bf5b-03138318b1c8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bc6a4c7-1a9d-4f6e-bf5b-03138318b1c8-maas-api-service-ca\") pod \"4bc6a4c7-1a9d-4f6e-bf5b-03138318b1c8\" (UID: \"4bc6a4c7-1a9d-4f6e-bf5b-03138318b1c8\") "
Oct 30 05:15:21.438572 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:21.438551    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bc6a4c7-1a9d-4f6e-bf5b-03138318b1c8-kube-api-access-cgjkk\" (UniqueName: \"kubernetes.io/projected/4bc6a4c7-1a9d-4f6e-bf5b-03138318b1c8-kube-api-access-cgjkk\") pod \"4bc6a4c7-1a9d-4f6e-bf5b-03138318b1c8\" (UID: \"4bc6a4c7-1a9d-4f6e-bf5b-03138318b1c8\") "
Oct 30 05:15:21.438629 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:21.438617    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8920d90-c187-4e68-a3a0-ff76a5f28d1b-kube-api-access-7l6tx\" (UniqueName: \"kubernetes.io/projected/d8920d90-c187-4e68-a3a0-ff76a5f28d1b-kube-api-access-7l6tx\") pod \"d8920d90-c187-4e68-a3a0-ff76a5f28d1b\" (UID: \"d8920d90-c187-4e68-a3a0-ff76a5f28d1b\") "
Oct 30 05:15:21.438746 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:21.438722    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8920d90-c187-4e68-a3a0-ff76a5f28d1b-maas-api-service-ca" pod "d8920d90-c187-4e68-a3a0-ff76a5f28d1b" (UID: "d8920d90-c187-4e68-a3a0-ff76a5f28d1b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:15:21.438817 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:21.438794    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bc6a4c7-1a9d-4f6e-bf5b-03138318b1c8-maas-api-service-ca" pod "4bc6a4c7-1a9d-4f6e-bf5b-03138318b1c8" (UID: "4bc6a4c7-1a9d-4f6e-bf5b-03138318b1c8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:15:21.439147 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:21.439127    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8920d90-c187-4e68-a3a0-ff76a5f28d1b-maas-api-service-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:15:21.439147 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:21.439146    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bc6a4c7-1a9d-4f6e-bf5b-03138318b1c8-maas-api-service-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:15:21.440834 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:21.440814    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8920d90-c187-4e68-a3a0-ff76a5f28d1b-kube-api-access-7l6tx" pod "d8920d90-c187-4e68-a3a0-ff76a5f28d1b" (UID: "d8920d90-c187-4e68-a3a0-ff76a5f28d1b"). InnerVolumeSpecName "kube-api-access-7l6tx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:15:21.441064 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:21.441044    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bc6a4c7-1a9d-4f6e-bf5b-03138318b1c8-kube-api-access-cgjkk" pod "4bc6a4c7-1a9d-4f6e-bf5b-03138318b1c8" (UID: "4bc6a4c7-1a9d-4f6e-bf5b-03138318b1c8"). InnerVolumeSpecName "kube-api-access-cgjkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:15:21.540335 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:21.540296    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgjkk\" (UniqueName: \"kubernetes.io/projected/4bc6a4c7-1a9d-4f6e-bf5b-03138318b1c8-kube-api-access-cgjkk\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:15:21.540335 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:21.540329    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l6tx\" (UniqueName: \"kubernetes.io/projected/d8920d90-c187-4e68-a3a0-ff76a5f28d1b-kube-api-access-7l6tx\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:15:22.075623 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:22.075595    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-010013-29888955-d4m2d"
Oct 30 05:15:22.076142 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:22.075636    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-010013-29888955-d4m2d" event={"ID":"4bc6a4c7-1a9d-4f6e-bf5b-03138318b1c8","Type":"ContainerDied","Data":"dbcce354f2215bb9664655e4099ccd5ce8b67d08cfdc0ba7b1fa0ff82c019d2d"}
Oct 30 05:15:22.076142 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:22.075697    5898 scope.go:122] "RemoveContainer" containerID="ed2ab0627f24cd88c5e77a72214cc93a4d1ddfb8062fd185e637644a0eeb2aac"
Oct 30 05:15:22.077115 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:22.077092    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c3486b-29888955-2n6bs" event={"ID":"d8920d90-c187-4e68-a3a0-ff76a5f28d1b","Type":"ContainerDied","Data":"30cbe54bbb7c0e7db064bf0f7036e2b8b5bd2a8837ada5459363e47ae36ab6a1"}
Oct 30 05:15:22.077206 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:22.077115    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c3486b-29888955-2n6bs"
Oct 30 05:15:22.087162 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:22.087139    5898 scope.go:122] "RemoveContainer" containerID="9da1d96e863b17f115fad4a152f522ca9365cea8f7c3b6c9beee8f6c6d154ff8"
Oct 30 05:15:22.134607 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:22.134444    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c3486b-29888955-2n6bs"]
Oct 30 05:15:22.144958 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:22.144930    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c3486b-29888955-2n6bs"]
Oct 30 05:15:22.169868 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:22.169834    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-010013-29888955-d4m2d"]
Oct 30 05:15:22.177942 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:22.177915    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-010013-29888955-d4m2d"]
Oct 30 05:15:22.892688 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:22.892651    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bc6a4c7-1a9d-4f6e-bf5b-03138318b1c8" path="/var/lib/kubelet/pods/4bc6a4c7-1a9d-4f6e-bf5b-03138318b1c8/volumes"
Oct 30 05:15:22.893156 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:22.893143    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71250a09-9976-4049-9280-de3ccdfd3ab0" path="/var/lib/kubelet/pods/71250a09-9976-4049-9280-de3ccdfd3ab0/volumes"
Oct 30 05:15:22.893541 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:22.893529    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb3a1f87-17d3-42d5-9019-78a8b6d48825" path="/var/lib/kubelet/pods/bb3a1f87-17d3-42d5-9019-78a8b6d48825/volumes"
Oct 30 05:15:22.894262 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:22.894246    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8920d90-c187-4e68-a3a0-ff76a5f28d1b" path="/var/lib/kubelet/pods/d8920d90-c187-4e68-a3a0-ff76a5f28d1b/volumes"
Oct 30 05:15:23.117717 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:23.117682    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:25.064529 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:25.064489    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:26.819376 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:15:26.819336    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:15:26.832657 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:26.832633    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:27.049928 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:27.049886    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-453511/test-model-453511-kserve-54dcc5c589-wmt2z"
Oct 30 05:15:27.063716 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:27.063688    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-453511/test-model-453511-kserve-54dcc5c589-wmt2z"
Oct 30 05:15:27.141557 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:27.141522    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-453511/test-model-453511-kserve-54dcc5c589-wmt2z"
Oct 30 05:15:28.523062 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:28.523023    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:30.073609 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:30.073573    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:30.169182 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:30.169140    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:30.474020 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:30.473944    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:32.038961 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:32.038928    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:33.821440 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:33.821402    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:35.719295 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:35.719260    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:36.654366 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:36.654327    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:37.120457 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.120423    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:37.590127 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.590086    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p"]
Oct 30 05:15:37.590826 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.590804    5898 cpu_manager.go:414] "containerMap: removing container" podUID="206ca9bd-dbed-464d-9bff-4a6195a3cda9" containerName="cleanup"
Oct 30 05:15:37.590826 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.590824    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="206ca9bd-dbed-464d-9bff-4a6195a3cda9" containerName="cleanup"
Oct 30 05:15:37.591022 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.590834    5898 cpu_manager.go:414] "containerMap: removing container" podUID="b453b1b2-fd20-4f49-ba72-9cc2588e4201" containerName="cleanup"
Oct 30 05:15:37.591022 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.590840    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b453b1b2-fd20-4f49-ba72-9cc2588e4201" containerName="cleanup"
Oct 30 05:15:37.591022 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.590850    5898 cpu_manager.go:414] "containerMap: removing container" podUID="b453b1b2-fd20-4f49-ba72-9cc2588e4201" containerName="cleanup"
Oct 30 05:15:37.591022 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.590855    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b453b1b2-fd20-4f49-ba72-9cc2588e4201" containerName="cleanup"
Oct 30 05:15:37.591022 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.590869    5898 cpu_manager.go:414] "containerMap: removing container" podUID="4bc6a4c7-1a9d-4f6e-bf5b-03138318b1c8" containerName="cleanup"
Oct 30 05:15:37.591022 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.590874    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bc6a4c7-1a9d-4f6e-bf5b-03138318b1c8" containerName="cleanup"
Oct 30 05:15:37.591022 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.590886    5898 cpu_manager.go:414] "containerMap: removing container" podUID="d8920d90-c187-4e68-a3a0-ff76a5f28d1b" containerName="cleanup"
Oct 30 05:15:37.591022 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.590891    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8920d90-c187-4e68-a3a0-ff76a5f28d1b" containerName="cleanup"
Oct 30 05:15:37.591022 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.590915    5898 cpu_manager.go:414] "containerMap: removing container" podUID="d8920d90-c187-4e68-a3a0-ff76a5f28d1b" containerName="cleanup"
Oct 30 05:15:37.591022 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.590922    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8920d90-c187-4e68-a3a0-ff76a5f28d1b" containerName="cleanup"
Oct 30 05:15:37.591022 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.590928    5898 cpu_manager.go:414] "containerMap: removing container" podUID="bb3a1f87-17d3-42d5-9019-78a8b6d48825" containerName="cleanup"
Oct 30 05:15:37.591022 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.590932    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb3a1f87-17d3-42d5-9019-78a8b6d48825" containerName="cleanup"
Oct 30 05:15:37.591022 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.590946    5898 cpu_manager.go:414] "containerMap: removing container" podUID="88933a2c-374f-4eb4-98ed-b7f3438774c3" containerName="cleanup"
Oct 30 05:15:37.591022 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.590951    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88933a2c-374f-4eb4-98ed-b7f3438774c3" containerName="cleanup"
Oct 30 05:15:37.591022 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.590965    5898 cpu_manager.go:414] "containerMap: removing container" podUID="938a8472-fb50-452a-bef7-416b671f3362" containerName="cleanup"
Oct 30 05:15:37.591022 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.590970    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="938a8472-fb50-452a-bef7-416b671f3362" containerName="cleanup"
Oct 30 05:15:37.591022 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.590977    5898 cpu_manager.go:414] "containerMap: removing container" podUID="3ee1e0d5-861e-41ac-929d-cfbdb7d75cd6" containerName="cleanup"
Oct 30 05:15:37.591022 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.590982    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ee1e0d5-861e-41ac-929d-cfbdb7d75cd6" containerName="cleanup"
Oct 30 05:15:37.591022 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.590991    5898 cpu_manager.go:414] "containerMap: removing container" podUID="88933a2c-374f-4eb4-98ed-b7f3438774c3" containerName="cleanup"
Oct 30 05:15:37.591022 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.590997    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88933a2c-374f-4eb4-98ed-b7f3438774c3" containerName="cleanup"
Oct 30 05:15:37.591022 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.591005    5898 cpu_manager.go:414] "containerMap: removing container" podUID="71250a09-9976-4049-9280-de3ccdfd3ab0" containerName="cleanup"
Oct 30 05:15:37.591022 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.591009    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71250a09-9976-4049-9280-de3ccdfd3ab0" containerName="cleanup"
Oct 30 05:15:37.591022 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.591016    5898 cpu_manager.go:414] "containerMap: removing container" podUID="938a8472-fb50-452a-bef7-416b671f3362" containerName="cleanup"
Oct 30 05:15:37.591022 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.591020    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="938a8472-fb50-452a-bef7-416b671f3362" containerName="cleanup"
Oct 30 05:15:37.591022 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.591027    5898 cpu_manager.go:414] "containerMap: removing container" podUID="938a8472-fb50-452a-bef7-416b671f3362" containerName="cleanup"
Oct 30 05:15:37.591022 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.591032    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="938a8472-fb50-452a-bef7-416b671f3362" containerName="cleanup"
Oct 30 05:15:37.591812 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.591039    5898 cpu_manager.go:414] "containerMap: removing container" podUID="71250a09-9976-4049-9280-de3ccdfd3ab0" containerName="cleanup"
Oct 30 05:15:37.591812 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.591044    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71250a09-9976-4049-9280-de3ccdfd3ab0" containerName="cleanup"
Oct 30 05:15:37.591812 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.591063    5898 cpu_manager.go:414] "containerMap: removing container" podUID="71250a09-9976-4049-9280-de3ccdfd3ab0" containerName="cleanup"
Oct 30 05:15:37.591812 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.591071    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71250a09-9976-4049-9280-de3ccdfd3ab0" containerName="cleanup"
Oct 30 05:15:37.591812 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.591078    5898 cpu_manager.go:414] "containerMap: removing container" podUID="206ca9bd-dbed-464d-9bff-4a6195a3cda9" containerName="cleanup"
Oct 30 05:15:37.591812 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.591082    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="206ca9bd-dbed-464d-9bff-4a6195a3cda9" containerName="cleanup"
Oct 30 05:15:37.591812 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.591091    5898 cpu_manager.go:414] "containerMap: removing container" podUID="bb3a1f87-17d3-42d5-9019-78a8b6d48825" containerName="cleanup"
Oct 30 05:15:37.591812 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.591095    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb3a1f87-17d3-42d5-9019-78a8b6d48825" containerName="cleanup"
Oct 30 05:15:37.591812 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.591103    5898 cpu_manager.go:414] "containerMap: removing container" podUID="3ee1e0d5-861e-41ac-929d-cfbdb7d75cd6" containerName="cleanup"
Oct 30 05:15:37.591812 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.591108    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ee1e0d5-861e-41ac-929d-cfbdb7d75cd6" containerName="cleanup"
Oct 30 05:15:37.591812 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.591115    5898 cpu_manager.go:414] "containerMap: removing container" podUID="3ee1e0d5-861e-41ac-929d-cfbdb7d75cd6" containerName="cleanup"
Oct 30 05:15:37.591812 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.591119    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ee1e0d5-861e-41ac-929d-cfbdb7d75cd6" containerName="cleanup"
Oct 30 05:15:37.591812 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.591126    5898 cpu_manager.go:414] "containerMap: removing container" podUID="4bc6a4c7-1a9d-4f6e-bf5b-03138318b1c8" containerName="cleanup"
Oct 30 05:15:37.591812 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.591130    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bc6a4c7-1a9d-4f6e-bf5b-03138318b1c8" containerName="cleanup"
Oct 30 05:15:37.591812 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.591239    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="b453b1b2-fd20-4f49-ba72-9cc2588e4201" containerName="cleanup"
Oct 30 05:15:37.591812 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.591247    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="b453b1b2-fd20-4f49-ba72-9cc2588e4201" containerName="cleanup"
Oct 30 05:15:37.591812 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.591254    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bc6a4c7-1a9d-4f6e-bf5b-03138318b1c8" containerName="cleanup"
Oct 30 05:15:37.591812 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.591262    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8920d90-c187-4e68-a3a0-ff76a5f28d1b" containerName="cleanup"
Oct 30 05:15:37.591812 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.591268    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8920d90-c187-4e68-a3a0-ff76a5f28d1b" containerName="cleanup"
Oct 30 05:15:37.591812 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.591275    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb3a1f87-17d3-42d5-9019-78a8b6d48825" containerName="cleanup"
Oct 30 05:15:37.591812 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.591282    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="88933a2c-374f-4eb4-98ed-b7f3438774c3" containerName="cleanup"
Oct 30 05:15:37.591812 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.591288    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb3a1f87-17d3-42d5-9019-78a8b6d48825" containerName="cleanup"
Oct 30 05:15:37.591812 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.591299    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="b453b1b2-fd20-4f49-ba72-9cc2588e4201" containerName="cleanup"
Oct 30 05:15:37.591812 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.591305    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="938a8472-fb50-452a-bef7-416b671f3362" containerName="cleanup"
Oct 30 05:15:37.591812 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.591311    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ee1e0d5-861e-41ac-929d-cfbdb7d75cd6" containerName="cleanup"
Oct 30 05:15:37.591812 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.591317    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="88933a2c-374f-4eb4-98ed-b7f3438774c3" containerName="cleanup"
Oct 30 05:15:37.591812 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.591323    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="71250a09-9976-4049-9280-de3ccdfd3ab0" containerName="cleanup"
Oct 30 05:15:37.591812 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.591328    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="938a8472-fb50-452a-bef7-416b671f3362" containerName="cleanup"
Oct 30 05:15:37.591812 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.591333    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="938a8472-fb50-452a-bef7-416b671f3362" containerName="cleanup"
Oct 30 05:15:37.591812 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.591339    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="71250a09-9976-4049-9280-de3ccdfd3ab0" containerName="cleanup"
Oct 30 05:15:37.591812 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.591345    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bc6a4c7-1a9d-4f6e-bf5b-03138318b1c8" containerName="cleanup"
Oct 30 05:15:37.591812 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.591351    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="71250a09-9976-4049-9280-de3ccdfd3ab0" containerName="cleanup"
Oct 30 05:15:37.591812 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.591357    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="206ca9bd-dbed-464d-9bff-4a6195a3cda9" containerName="cleanup"
Oct 30 05:15:37.591812 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.591363    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="206ca9bd-dbed-464d-9bff-4a6195a3cda9" containerName="cleanup"
Oct 30 05:15:37.591812 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.591372    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ee1e0d5-861e-41ac-929d-cfbdb7d75cd6" containerName="cleanup"
Oct 30 05:15:37.591812 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.591380    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ee1e0d5-861e-41ac-929d-cfbdb7d75cd6" containerName="cleanup"
Oct 30 05:15:37.597078 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.597057    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p"
Oct 30 05:15:37.623142 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.620873    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p"]
Oct 30 05:15:37.624180 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.624148    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-2a8f18-openshift-default-dockercfg-c6srh\""
Oct 30 05:15:37.648824 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.648791    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:37.663004 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.662969    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1d94e3b-c651-4989-afbe-53acbd131f71-istio-token\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p\" (UID: \"e1d94e3b-c651-4989-afbe-53acbd131f71\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p"
Oct 30 05:15:37.663209 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.663029    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1d94e3b-c651-4989-afbe-53acbd131f71-istio-podinfo\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p\" (UID: \"e1d94e3b-c651-4989-afbe-53acbd131f71\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p"
Oct 30 05:15:37.663209 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.663160    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1d94e3b-c651-4989-afbe-53acbd131f71-istio-data\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p\" (UID: \"e1d94e3b-c651-4989-afbe-53acbd131f71\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p"
Oct 30 05:15:37.663209 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.663205    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1d94e3b-c651-4989-afbe-53acbd131f71-workload-socket\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p\" (UID: \"e1d94e3b-c651-4989-afbe-53acbd131f71\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p"
Oct 30 05:15:37.663390 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.663228    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1d94e3b-c651-4989-afbe-53acbd131f71-istio-envoy\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p\" (UID: \"e1d94e3b-c651-4989-afbe-53acbd131f71\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p"
Oct 30 05:15:37.663390 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.663268    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1d94e3b-c651-4989-afbe-53acbd131f71-workload-certs\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p\" (UID: \"e1d94e3b-c651-4989-afbe-53acbd131f71\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p"
Oct 30 05:15:37.663390 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.663291    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbpxd\" (UniqueName: \"kubernetes.io/projected/e1d94e3b-c651-4989-afbe-53acbd131f71-kube-api-access-pbpxd\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p\" (UID: \"e1d94e3b-c651-4989-afbe-53acbd131f71\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p"
Oct 30 05:15:37.663390 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.663313    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1d94e3b-c651-4989-afbe-53acbd131f71-istiod-ca-cert\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p\" (UID: \"e1d94e3b-c651-4989-afbe-53acbd131f71\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p"
Oct 30 05:15:37.663390 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.663346    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1d94e3b-c651-4989-afbe-53acbd131f71-credential-socket\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p\" (UID: \"e1d94e3b-c651-4989-afbe-53acbd131f71\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p"
Oct 30 05:15:37.768032 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.767980    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1d94e3b-c651-4989-afbe-53acbd131f71-istio-data\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p\" (UID: \"e1d94e3b-c651-4989-afbe-53acbd131f71\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p"
Oct 30 05:15:37.768329 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.768307    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1d94e3b-c651-4989-afbe-53acbd131f71-workload-socket\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p\" (UID: \"e1d94e3b-c651-4989-afbe-53acbd131f71\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p"
Oct 30 05:15:37.768458 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.768443    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1d94e3b-c651-4989-afbe-53acbd131f71-istio-envoy\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p\" (UID: \"e1d94e3b-c651-4989-afbe-53acbd131f71\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p"
Oct 30 05:15:37.768602 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.768588    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1d94e3b-c651-4989-afbe-53acbd131f71-workload-certs\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p\" (UID: \"e1d94e3b-c651-4989-afbe-53acbd131f71\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p"
Oct 30 05:15:37.768717 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.768703    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbpxd\" (UniqueName: \"kubernetes.io/projected/e1d94e3b-c651-4989-afbe-53acbd131f71-kube-api-access-pbpxd\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p\" (UID: \"e1d94e3b-c651-4989-afbe-53acbd131f71\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p"
Oct 30 05:15:37.768836 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.768822    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1d94e3b-c651-4989-afbe-53acbd131f71-istiod-ca-cert\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p\" (UID: \"e1d94e3b-c651-4989-afbe-53acbd131f71\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p"
Oct 30 05:15:37.768999 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.768981    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1d94e3b-c651-4989-afbe-53acbd131f71-credential-socket\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p\" (UID: \"e1d94e3b-c651-4989-afbe-53acbd131f71\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p"
Oct 30 05:15:37.769293 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.769274    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1d94e3b-c651-4989-afbe-53acbd131f71-istio-token\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p\" (UID: \"e1d94e3b-c651-4989-afbe-53acbd131f71\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p"
Oct 30 05:15:37.769426 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.769409    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1d94e3b-c651-4989-afbe-53acbd131f71-istio-podinfo\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p\" (UID: \"e1d94e3b-c651-4989-afbe-53acbd131f71\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p"
Oct 30 05:15:37.782328 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.776374    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1d94e3b-c651-4989-afbe-53acbd131f71-workload-certs\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p\" (UID: \"e1d94e3b-c651-4989-afbe-53acbd131f71\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p"
Oct 30 05:15:37.782609 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.781598    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1d94e3b-c651-4989-afbe-53acbd131f71-istio-podinfo\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p\" (UID: \"e1d94e3b-c651-4989-afbe-53acbd131f71\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p"
Oct 30 05:15:37.782747 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.782656    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1d94e3b-c651-4989-afbe-53acbd131f71-istiod-ca-cert\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p\" (UID: \"e1d94e3b-c651-4989-afbe-53acbd131f71\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p"
Oct 30 05:15:37.783192 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.783170    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1d94e3b-c651-4989-afbe-53acbd131f71-istio-data\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p\" (UID: \"e1d94e3b-c651-4989-afbe-53acbd131f71\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p"
Oct 30 05:15:37.807249 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.783837    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1d94e3b-c651-4989-afbe-53acbd131f71-workload-socket\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p\" (UID: \"e1d94e3b-c651-4989-afbe-53acbd131f71\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p"
Oct 30 05:15:37.807480 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.782882    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1d94e3b-c651-4989-afbe-53acbd131f71-credential-socket\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p\" (UID: \"e1d94e3b-c651-4989-afbe-53acbd131f71\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p"
Oct 30 05:15:37.807621 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.794371    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:37.807800 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.807755    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbpxd\" (UniqueName: \"kubernetes.io/projected/e1d94e3b-c651-4989-afbe-53acbd131f71-kube-api-access-pbpxd\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p\" (UID: \"e1d94e3b-c651-4989-afbe-53acbd131f71\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p"
Oct 30 05:15:37.807888 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.795607    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1d94e3b-c651-4989-afbe-53acbd131f71-istio-envoy\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p\" (UID: \"e1d94e3b-c651-4989-afbe-53acbd131f71\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p"
Oct 30 05:15:37.841973 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.841328    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1d94e3b-c651-4989-afbe-53acbd131f71-istio-token\") pod \"e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p\" (UID: \"e1d94e3b-c651-4989-afbe-53acbd131f71\") " pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p"
Oct 30 05:15:37.910718 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.910582    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:37.930504 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:37.930198    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p"
Oct 30 05:15:38.323215 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:38.323184    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p"]
Oct 30 05:15:38.326711 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:15:38.326681    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1d94e3b_c651_4989_afbe_53acbd131f71.slice/crio-613529a61e138694536993792943b1e98c6a6410993dd56960db9d88c1532a11 WatchSource:0}: Error finding container 613529a61e138694536993792943b1e98c6a6410993dd56960db9d88c1532a11: Status 404 returned error can't find the container with id 613529a61e138694536993792943b1e98c6a6410993dd56960db9d88c1532a11
Oct 30 05:15:38.329580 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:38.329521    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:15:38.329669 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:38.329611    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:15:38.329669 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:38.329644    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:15:39.227090 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:39.227048    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p" event={"ID":"e1d94e3b-c651-4989-afbe-53acbd131f71","Type":"ContainerStarted","Data":"d56328b64cfc56d88c83488b4f04061781785f631a042f33f74b30d94e1dae95"}
Oct 30 05:15:39.227298 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:39.227117    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p" event={"ID":"e1d94e3b-c651-4989-afbe-53acbd131f71","Type":"ContainerStarted","Data":"613529a61e138694536993792943b1e98c6a6410993dd56960db9d88c1532a11"}
Oct 30 05:15:39.291911 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:39.291849    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p" podStartSLOduration=2.291835827 podStartE2EDuration="2.291835827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:15:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:15:39.277594226 +0000 UTC m=+1868.501414128" watchObservedRunningTime="2026-10-30 05:15:39.291835827 +0000 UTC m=+1868.515655798"
Oct 30 05:15:39.618924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:39.617983    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-36fd3c/test-model-36fd3c-kserve-86d45f7b9c-qhhbc"]
Oct 30 05:15:39.618924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:39.618893    5898 cpu_manager.go:414] "containerMap: removing container" podUID="206ca9bd-dbed-464d-9bff-4a6195a3cda9" containerName="cleanup"
Oct 30 05:15:39.618924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:39.618928    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="206ca9bd-dbed-464d-9bff-4a6195a3cda9" containerName="cleanup"
Oct 30 05:15:39.619548 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:39.618948    5898 cpu_manager.go:414] "containerMap: removing container" podUID="b453b1b2-fd20-4f49-ba72-9cc2588e4201" containerName="cleanup"
Oct 30 05:15:39.619548 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:39.618956    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b453b1b2-fd20-4f49-ba72-9cc2588e4201" containerName="cleanup"
Oct 30 05:15:39.619548 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:39.618978    5898 cpu_manager.go:414] "containerMap: removing container" podUID="4bc6a4c7-1a9d-4f6e-bf5b-03138318b1c8" containerName="cleanup"
Oct 30 05:15:39.619548 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:39.618986    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bc6a4c7-1a9d-4f6e-bf5b-03138318b1c8" containerName="cleanup"
Oct 30 05:15:39.619548 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:39.619003    5898 cpu_manager.go:414] "containerMap: removing container" podUID="d8920d90-c187-4e68-a3a0-ff76a5f28d1b" containerName="cleanup"
Oct 30 05:15:39.619548 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:39.619011    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8920d90-c187-4e68-a3a0-ff76a5f28d1b" containerName="cleanup"
Oct 30 05:15:39.619548 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:39.619024    5898 cpu_manager.go:414] "containerMap: removing container" podUID="bb3a1f87-17d3-42d5-9019-78a8b6d48825" containerName="cleanup"
Oct 30 05:15:39.619548 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:39.619031    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb3a1f87-17d3-42d5-9019-78a8b6d48825" containerName="cleanup"
Oct 30 05:15:39.619548 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:39.619048    5898 cpu_manager.go:414] "containerMap: removing container" podUID="88933a2c-374f-4eb4-98ed-b7f3438774c3" containerName="cleanup"
Oct 30 05:15:39.619548 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:39.619055    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88933a2c-374f-4eb4-98ed-b7f3438774c3" containerName="cleanup"
Oct 30 05:15:39.619548 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:39.619235    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8920d90-c187-4e68-a3a0-ff76a5f28d1b" containerName="cleanup"
Oct 30 05:15:39.619548 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:39.619252    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb3a1f87-17d3-42d5-9019-78a8b6d48825" containerName="cleanup"
Oct 30 05:15:39.619548 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:39.619263    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="88933a2c-374f-4eb4-98ed-b7f3438774c3" containerName="cleanup"
Oct 30 05:15:39.619548 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:39.619274    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bc6a4c7-1a9d-4f6e-bf5b-03138318b1c8" containerName="cleanup"
Oct 30 05:15:39.619548 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:39.619289    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="206ca9bd-dbed-464d-9bff-4a6195a3cda9" containerName="cleanup"
Oct 30 05:15:39.635955 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:39.623650    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-36fd3c/test-model-36fd3c-kserve-86d45f7b9c-qhhbc"
Oct 30 05:15:39.672303 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:39.664881    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-36fd3c\"/\"kube-root-ca.crt\""
Oct 30 05:15:39.672303 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:39.665016    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-36fd3c\"/\"openshift-service-ca.crt\""
Oct 30 05:15:39.672303 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:39.669846    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-36fd3c/test-model-36fd3c-kserve-86d45f7b9c-qhhbc"]
Oct 30 05:15:39.677944 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:39.675989    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-36fd3c\"/\"default-dockercfg-6dng2\""
Oct 30 05:15:39.677944 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:39.676064    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-36fd3c\"/\"test-model-36fd3c-kserve-self-signed-certs\""
Oct 30 05:15:39.677944 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:39.677252    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:39.691856 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:39.691821    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmb5j\" (UniqueName: \"kubernetes.io/projected/da336e19-5be4-46ea-80c9-1541cacfa7f0-kube-api-access-cmb5j\") pod \"test-model-36fd3c-kserve-86d45f7b9c-qhhbc\" (UID: \"da336e19-5be4-46ea-80c9-1541cacfa7f0\") " pod="ai-tenant-e2e-inf-b-36fd3c/test-model-36fd3c-kserve-86d45f7b9c-qhhbc"
Oct 30 05:15:39.692100 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:39.692073    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da336e19-5be4-46ea-80c9-1541cacfa7f0-tls-certs\") pod \"test-model-36fd3c-kserve-86d45f7b9c-qhhbc\" (UID: \"da336e19-5be4-46ea-80c9-1541cacfa7f0\") " pod="ai-tenant-e2e-inf-b-36fd3c/test-model-36fd3c-kserve-86d45f7b9c-qhhbc"
Oct 30 05:15:39.692182 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:39.692131    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da336e19-5be4-46ea-80c9-1541cacfa7f0-tmp-dir\") pod \"test-model-36fd3c-kserve-86d45f7b9c-qhhbc\" (UID: \"da336e19-5be4-46ea-80c9-1541cacfa7f0\") " pod="ai-tenant-e2e-inf-b-36fd3c/test-model-36fd3c-kserve-86d45f7b9c-qhhbc"
Oct 30 05:15:39.692247 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:39.692227    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da336e19-5be4-46ea-80c9-1541cacfa7f0-dshm\") pod \"test-model-36fd3c-kserve-86d45f7b9c-qhhbc\" (UID: \"da336e19-5be4-46ea-80c9-1541cacfa7f0\") " pod="ai-tenant-e2e-inf-b-36fd3c/test-model-36fd3c-kserve-86d45f7b9c-qhhbc"
Oct 30 05:15:39.692318 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:39.692302    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da336e19-5be4-46ea-80c9-1541cacfa7f0-home\") pod \"test-model-36fd3c-kserve-86d45f7b9c-qhhbc\" (UID: \"da336e19-5be4-46ea-80c9-1541cacfa7f0\") " pod="ai-tenant-e2e-inf-b-36fd3c/test-model-36fd3c-kserve-86d45f7b9c-qhhbc"
Oct 30 05:15:39.692379 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:39.692334    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da336e19-5be4-46ea-80c9-1541cacfa7f0-model-cache\") pod \"test-model-36fd3c-kserve-86d45f7b9c-qhhbc\" (UID: \"da336e19-5be4-46ea-80c9-1541cacfa7f0\") " pod="ai-tenant-e2e-inf-b-36fd3c/test-model-36fd3c-kserve-86d45f7b9c-qhhbc"
Oct 30 05:15:39.793488 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:39.793443    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmb5j\" (UniqueName: \"kubernetes.io/projected/da336e19-5be4-46ea-80c9-1541cacfa7f0-kube-api-access-cmb5j\") pod \"test-model-36fd3c-kserve-86d45f7b9c-qhhbc\" (UID: \"da336e19-5be4-46ea-80c9-1541cacfa7f0\") " pod="ai-tenant-e2e-inf-b-36fd3c/test-model-36fd3c-kserve-86d45f7b9c-qhhbc"
Oct 30 05:15:39.793672 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:39.793603    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da336e19-5be4-46ea-80c9-1541cacfa7f0-tls-certs\") pod \"test-model-36fd3c-kserve-86d45f7b9c-qhhbc\" (UID: \"da336e19-5be4-46ea-80c9-1541cacfa7f0\") " pod="ai-tenant-e2e-inf-b-36fd3c/test-model-36fd3c-kserve-86d45f7b9c-qhhbc"
Oct 30 05:15:39.793672 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:39.793651    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da336e19-5be4-46ea-80c9-1541cacfa7f0-tmp-dir\") pod \"test-model-36fd3c-kserve-86d45f7b9c-qhhbc\" (UID: \"da336e19-5be4-46ea-80c9-1541cacfa7f0\") " pod="ai-tenant-e2e-inf-b-36fd3c/test-model-36fd3c-kserve-86d45f7b9c-qhhbc"
Oct 30 05:15:39.793787 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:39.793751    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da336e19-5be4-46ea-80c9-1541cacfa7f0-dshm\") pod \"test-model-36fd3c-kserve-86d45f7b9c-qhhbc\" (UID: \"da336e19-5be4-46ea-80c9-1541cacfa7f0\") " pod="ai-tenant-e2e-inf-b-36fd3c/test-model-36fd3c-kserve-86d45f7b9c-qhhbc"
Oct 30 05:15:39.793843 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:39.793827    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da336e19-5be4-46ea-80c9-1541cacfa7f0-home\") pod \"test-model-36fd3c-kserve-86d45f7b9c-qhhbc\" (UID: \"da336e19-5be4-46ea-80c9-1541cacfa7f0\") " pod="ai-tenant-e2e-inf-b-36fd3c/test-model-36fd3c-kserve-86d45f7b9c-qhhbc"
Oct 30 05:15:39.793936 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:39.793853    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da336e19-5be4-46ea-80c9-1541cacfa7f0-model-cache\") pod \"test-model-36fd3c-kserve-86d45f7b9c-qhhbc\" (UID: \"da336e19-5be4-46ea-80c9-1541cacfa7f0\") " pod="ai-tenant-e2e-inf-b-36fd3c/test-model-36fd3c-kserve-86d45f7b9c-qhhbc"
Oct 30 05:15:39.794239 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:39.794212    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da336e19-5be4-46ea-80c9-1541cacfa7f0-tmp-dir\") pod \"test-model-36fd3c-kserve-86d45f7b9c-qhhbc\" (UID: \"da336e19-5be4-46ea-80c9-1541cacfa7f0\") " pod="ai-tenant-e2e-inf-b-36fd3c/test-model-36fd3c-kserve-86d45f7b9c-qhhbc"
Oct 30 05:15:39.794334 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:39.794295    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da336e19-5be4-46ea-80c9-1541cacfa7f0-home\") pod \"test-model-36fd3c-kserve-86d45f7b9c-qhhbc\" (UID: \"da336e19-5be4-46ea-80c9-1541cacfa7f0\") " pod="ai-tenant-e2e-inf-b-36fd3c/test-model-36fd3c-kserve-86d45f7b9c-qhhbc"
Oct 30 05:15:39.794437 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:39.794410    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da336e19-5be4-46ea-80c9-1541cacfa7f0-model-cache\") pod \"test-model-36fd3c-kserve-86d45f7b9c-qhhbc\" (UID: \"da336e19-5be4-46ea-80c9-1541cacfa7f0\") " pod="ai-tenant-e2e-inf-b-36fd3c/test-model-36fd3c-kserve-86d45f7b9c-qhhbc"
Oct 30 05:15:39.796726 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:39.796697    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da336e19-5be4-46ea-80c9-1541cacfa7f0-dshm\") pod \"test-model-36fd3c-kserve-86d45f7b9c-qhhbc\" (UID: \"da336e19-5be4-46ea-80c9-1541cacfa7f0\") " pod="ai-tenant-e2e-inf-b-36fd3c/test-model-36fd3c-kserve-86d45f7b9c-qhhbc"
Oct 30 05:15:39.797014 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:39.796996    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da336e19-5be4-46ea-80c9-1541cacfa7f0-tls-certs\") pod \"test-model-36fd3c-kserve-86d45f7b9c-qhhbc\" (UID: \"da336e19-5be4-46ea-80c9-1541cacfa7f0\") " pod="ai-tenant-e2e-inf-b-36fd3c/test-model-36fd3c-kserve-86d45f7b9c-qhhbc"
Oct 30 05:15:39.811711 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:39.811689    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmb5j\" (UniqueName: \"kubernetes.io/projected/da336e19-5be4-46ea-80c9-1541cacfa7f0-kube-api-access-cmb5j\") pod \"test-model-36fd3c-kserve-86d45f7b9c-qhhbc\" (UID: \"da336e19-5be4-46ea-80c9-1541cacfa7f0\") " pod="ai-tenant-e2e-inf-b-36fd3c/test-model-36fd3c-kserve-86d45f7b9c-qhhbc"
Oct 30 05:15:39.844501 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:39.844472    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:39.942996 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:39.942483    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p"
Oct 30 05:15:39.979925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:39.979238    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-36fd3c/test-model-36fd3c-kserve-86d45f7b9c-qhhbc"
Oct 30 05:15:39.988442 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:39.987197    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p"
Oct 30 05:15:40.235038 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:40.234953    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p"
Oct 30 05:15:40.236032 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:40.236008    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p"
Oct 30 05:15:40.335423 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:40.335217    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-36fd3c/test-model-36fd3c-kserve-86d45f7b9c-qhhbc"]
Oct 30 05:15:40.345057 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:15:40.345020    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda336e19_5be4_46ea_80c9_1541cacfa7f0.slice/crio-f3eb37376825b6427b4fe546ee1ba0df45dcf69bfbfb31ca82f835a39c75802e WatchSource:0}: Error finding container f3eb37376825b6427b4fe546ee1ba0df45dcf69bfbfb31ca82f835a39c75802e: Status 404 returned error can't find the container with id f3eb37376825b6427b4fe546ee1ba0df45dcf69bfbfb31ca82f835a39c75802e
Oct 30 05:15:40.487990 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:40.487896    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:41.094674 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:41.094622    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:41.194042 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:41.194009    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:41.244107 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:41.244070    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-36fd3c/test-model-36fd3c-kserve-86d45f7b9c-qhhbc" event={"ID":"da336e19-5be4-46ea-80c9-1541cacfa7f0","Type":"ContainerStarted","Data":"9b049094af0e563a7941c8d609d7635c3ffdcf2cca5c3ce34c3a1fea7572d78f"}
Oct 30 05:15:41.244353 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:41.244122    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-36fd3c/test-model-36fd3c-kserve-86d45f7b9c-qhhbc" event={"ID":"da336e19-5be4-46ea-80c9-1541cacfa7f0","Type":"ContainerStarted","Data":"f3eb37376825b6427b4fe546ee1ba0df45dcf69bfbfb31ca82f835a39c75802e"}
Oct 30 05:15:41.283097 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:41.283039    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-36fd3c/test-model-36fd3c-kserve-86d45f7b9c-qhhbc" podStartSLOduration=2.283021777 podStartE2EDuration="2.283021777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:15:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:15:41.281993517 +0000 UTC m=+1870.505813437" watchObservedRunningTime="2026-10-30 05:15:41.283021777 +0000 UTC m=+1870.506841678"
Oct 30 05:15:41.694284 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:41.694251    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:41.795285 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:41.795252    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:41.901746 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:41.901714    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:41.980695 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:41.980616    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-36fd3c/test-model-36fd3c-kserve-86d45f7b9c-qhhbc"
Oct 30 05:15:42.000786 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:42.000759    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:43.490249 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:43.490214    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:43.590063 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:43.590004    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:43.816926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:43.816814    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:45.590841 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:45.590807    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:47.559024 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:47.558987    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:48.436679 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:48.436647    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-010013/test-model-010013-kserve-6bdfbb6f89-9fqsj"]
Oct 30 05:15:48.441507 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:48.441476    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-010013/test-model-010013-kserve-6bdfbb6f89-9fqsj"
Oct 30 05:15:48.460390 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:48.460361    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-010013/test-model-010013-kserve-6bdfbb6f89-9fqsj"]
Oct 30 05:15:48.466280 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:48.466250    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-010013\"/\"kube-root-ca.crt\""
Oct 30 05:15:48.466280 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:48.466250    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-010013\"/\"openshift-service-ca.crt\""
Oct 30 05:15:48.466474 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:48.466358    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-010013\"/\"default-dockercfg-m9ln6\""
Oct 30 05:15:48.468670 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:48.468622    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-010013\"/\"test-model-010013-kserve-self-signed-certs\""
Oct 30 05:15:48.495156 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:48.495129    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:48.546655 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:48.546623    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef140ba8-eb01-40bd-9b95-216c8bacd7af-dshm\") pod \"test-model-010013-kserve-6bdfbb6f89-9fqsj\" (UID: \"ef140ba8-eb01-40bd-9b95-216c8bacd7af\") " pod="ai-tenant-e2e-inf-a-010013/test-model-010013-kserve-6bdfbb6f89-9fqsj"
Oct 30 05:15:48.546827 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:48.546672    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef140ba8-eb01-40bd-9b95-216c8bacd7af-model-cache\") pod \"test-model-010013-kserve-6bdfbb6f89-9fqsj\" (UID: \"ef140ba8-eb01-40bd-9b95-216c8bacd7af\") " pod="ai-tenant-e2e-inf-a-010013/test-model-010013-kserve-6bdfbb6f89-9fqsj"
Oct 30 05:15:48.546827 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:48.546753    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef140ba8-eb01-40bd-9b95-216c8bacd7af-tmp-dir\") pod \"test-model-010013-kserve-6bdfbb6f89-9fqsj\" (UID: \"ef140ba8-eb01-40bd-9b95-216c8bacd7af\") " pod="ai-tenant-e2e-inf-a-010013/test-model-010013-kserve-6bdfbb6f89-9fqsj"
Oct 30 05:15:48.547021 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:48.546933    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ddjs\" (UniqueName: \"kubernetes.io/projected/ef140ba8-eb01-40bd-9b95-216c8bacd7af-kube-api-access-6ddjs\") pod \"test-model-010013-kserve-6bdfbb6f89-9fqsj\" (UID: \"ef140ba8-eb01-40bd-9b95-216c8bacd7af\") " pod="ai-tenant-e2e-inf-a-010013/test-model-010013-kserve-6bdfbb6f89-9fqsj"
Oct 30 05:15:48.547125 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:48.547097    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef140ba8-eb01-40bd-9b95-216c8bacd7af-tls-certs\") pod \"test-model-010013-kserve-6bdfbb6f89-9fqsj\" (UID: \"ef140ba8-eb01-40bd-9b95-216c8bacd7af\") " pod="ai-tenant-e2e-inf-a-010013/test-model-010013-kserve-6bdfbb6f89-9fqsj"
Oct 30 05:15:48.547194 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:48.547162    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef140ba8-eb01-40bd-9b95-216c8bacd7af-home\") pod \"test-model-010013-kserve-6bdfbb6f89-9fqsj\" (UID: \"ef140ba8-eb01-40bd-9b95-216c8bacd7af\") " pod="ai-tenant-e2e-inf-a-010013/test-model-010013-kserve-6bdfbb6f89-9fqsj"
Oct 30 05:15:48.648251 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:48.648214    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef140ba8-eb01-40bd-9b95-216c8bacd7af-home\") pod \"test-model-010013-kserve-6bdfbb6f89-9fqsj\" (UID: \"ef140ba8-eb01-40bd-9b95-216c8bacd7af\") " pod="ai-tenant-e2e-inf-a-010013/test-model-010013-kserve-6bdfbb6f89-9fqsj"
Oct 30 05:15:48.648744 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:48.648282    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef140ba8-eb01-40bd-9b95-216c8bacd7af-dshm\") pod \"test-model-010013-kserve-6bdfbb6f89-9fqsj\" (UID: \"ef140ba8-eb01-40bd-9b95-216c8bacd7af\") " pod="ai-tenant-e2e-inf-a-010013/test-model-010013-kserve-6bdfbb6f89-9fqsj"
Oct 30 05:15:48.648744 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:48.648312    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef140ba8-eb01-40bd-9b95-216c8bacd7af-model-cache\") pod \"test-model-010013-kserve-6bdfbb6f89-9fqsj\" (UID: \"ef140ba8-eb01-40bd-9b95-216c8bacd7af\") " pod="ai-tenant-e2e-inf-a-010013/test-model-010013-kserve-6bdfbb6f89-9fqsj"
Oct 30 05:15:48.648744 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:48.648346    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef140ba8-eb01-40bd-9b95-216c8bacd7af-tmp-dir\") pod \"test-model-010013-kserve-6bdfbb6f89-9fqsj\" (UID: \"ef140ba8-eb01-40bd-9b95-216c8bacd7af\") " pod="ai-tenant-e2e-inf-a-010013/test-model-010013-kserve-6bdfbb6f89-9fqsj"
Oct 30 05:15:48.648744 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:48.648447    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ddjs\" (UniqueName: \"kubernetes.io/projected/ef140ba8-eb01-40bd-9b95-216c8bacd7af-kube-api-access-6ddjs\") pod \"test-model-010013-kserve-6bdfbb6f89-9fqsj\" (UID: \"ef140ba8-eb01-40bd-9b95-216c8bacd7af\") " pod="ai-tenant-e2e-inf-a-010013/test-model-010013-kserve-6bdfbb6f89-9fqsj"
Oct 30 05:15:48.648744 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:48.648573    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef140ba8-eb01-40bd-9b95-216c8bacd7af-tls-certs\") pod \"test-model-010013-kserve-6bdfbb6f89-9fqsj\" (UID: \"ef140ba8-eb01-40bd-9b95-216c8bacd7af\") " pod="ai-tenant-e2e-inf-a-010013/test-model-010013-kserve-6bdfbb6f89-9fqsj"
Oct 30 05:15:48.648744 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:48.648653    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef140ba8-eb01-40bd-9b95-216c8bacd7af-home\") pod \"test-model-010013-kserve-6bdfbb6f89-9fqsj\" (UID: \"ef140ba8-eb01-40bd-9b95-216c8bacd7af\") " pod="ai-tenant-e2e-inf-a-010013/test-model-010013-kserve-6bdfbb6f89-9fqsj"
Oct 30 05:15:48.648744 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:48.648710    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef140ba8-eb01-40bd-9b95-216c8bacd7af-model-cache\") pod \"test-model-010013-kserve-6bdfbb6f89-9fqsj\" (UID: \"ef140ba8-eb01-40bd-9b95-216c8bacd7af\") " pod="ai-tenant-e2e-inf-a-010013/test-model-010013-kserve-6bdfbb6f89-9fqsj"
Oct 30 05:15:48.649177 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:48.648804    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef140ba8-eb01-40bd-9b95-216c8bacd7af-tmp-dir\") pod \"test-model-010013-kserve-6bdfbb6f89-9fqsj\" (UID: \"ef140ba8-eb01-40bd-9b95-216c8bacd7af\") " pod="ai-tenant-e2e-inf-a-010013/test-model-010013-kserve-6bdfbb6f89-9fqsj"
Oct 30 05:15:48.650657 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:48.650630    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef140ba8-eb01-40bd-9b95-216c8bacd7af-dshm\") pod \"test-model-010013-kserve-6bdfbb6f89-9fqsj\" (UID: \"ef140ba8-eb01-40bd-9b95-216c8bacd7af\") " pod="ai-tenant-e2e-inf-a-010013/test-model-010013-kserve-6bdfbb6f89-9fqsj"
Oct 30 05:15:48.650986 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:48.650967    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef140ba8-eb01-40bd-9b95-216c8bacd7af-tls-certs\") pod \"test-model-010013-kserve-6bdfbb6f89-9fqsj\" (UID: \"ef140ba8-eb01-40bd-9b95-216c8bacd7af\") " pod="ai-tenant-e2e-inf-a-010013/test-model-010013-kserve-6bdfbb6f89-9fqsj"
Oct 30 05:15:48.705975 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:48.703609    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ddjs\" (UniqueName: \"kubernetes.io/projected/ef140ba8-eb01-40bd-9b95-216c8bacd7af-kube-api-access-6ddjs\") pod \"test-model-010013-kserve-6bdfbb6f89-9fqsj\" (UID: \"ef140ba8-eb01-40bd-9b95-216c8bacd7af\") " pod="ai-tenant-e2e-inf-a-010013/test-model-010013-kserve-6bdfbb6f89-9fqsj"
Oct 30 05:15:48.761066 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:48.759672    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-010013/test-model-010013-kserve-6bdfbb6f89-9fqsj"
Oct 30 05:15:49.118726 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:49.118690    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-010013/test-model-010013-kserve-6bdfbb6f89-9fqsj"]
Oct 30 05:15:49.119240 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:15:49.119216    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podef140ba8_eb01_40bd_9b95_216c8bacd7af.slice/crio-5fa8a5509218bb2848c26b75943a5cbfdb57607a5d833c519e36e009be071608 WatchSource:0}: Error finding container 5fa8a5509218bb2848c26b75943a5cbfdb57607a5d833c519e36e009be071608: Status 404 returned error can't find the container with id 5fa8a5509218bb2848c26b75943a5cbfdb57607a5d833c519e36e009be071608
Oct 30 05:15:49.301688 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:49.301650    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-010013/test-model-010013-kserve-6bdfbb6f89-9fqsj" event={"ID":"ef140ba8-eb01-40bd-9b95-216c8bacd7af","Type":"ContainerStarted","Data":"cf4394eedfdb753f250e79e543329a93e7b403a9cde63821c85f068845e4a26a"}
Oct 30 05:15:49.301858 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:49.301701    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-010013/test-model-010013-kserve-6bdfbb6f89-9fqsj" event={"ID":"ef140ba8-eb01-40bd-9b95-216c8bacd7af","Type":"ContainerStarted","Data":"5fa8a5509218bb2848c26b75943a5cbfdb57607a5d833c519e36e009be071608"}
Oct 30 05:15:49.317020 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:49.316990    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:49.340524 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:49.340474    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-010013/test-model-010013-kserve-6bdfbb6f89-9fqsj" podStartSLOduration=1.340457211 podStartE2EDuration="1.340457211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:15:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:15:49.338532618 +0000 UTC m=+1878.562352518" watchObservedRunningTime="2026-10-30 05:15:49.340457211 +0000 UTC m=+1878.564277113"
Oct 30 05:15:49.760450 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:49.760420    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-010013/test-model-010013-kserve-6bdfbb6f89-9fqsj"
Oct 30 05:15:49.980514 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:49.980486    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-36fd3c/test-model-36fd3c-kserve-86d45f7b9c-qhhbc"
Oct 30 05:15:49.992735 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:49.992713    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-36fd3c/test-model-36fd3c-kserve-86d45f7b9c-qhhbc"
Oct 30 05:15:50.319503 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:50.319468    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-36fd3c/test-model-36fd3c-kserve-86d45f7b9c-qhhbc"
Oct 30 05:15:50.896006 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:50.895974    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:50.996637 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:50.996607    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:51.102838 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:51.102803    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:52.941601 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:52.941566    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:54.837811 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:54.837779    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:54.928184 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:54.928153    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:55.028977 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:55.028943    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:56.626345 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:56.626314    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:56.820090 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:15:56.820051    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:15:56.886646 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:56.886561    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:57.314169 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:57.314085    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:57.788705 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:57.788642    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:57.921126 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:57.921090    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:58.524334 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:58.524289    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:15:58.760608 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:58.760560    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-010013/test-model-010013-kserve-6bdfbb6f89-9fqsj"
Oct 30 05:15:58.778041 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:58.777969    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-010013/test-model-010013-kserve-6bdfbb6f89-9fqsj"
Oct 30 05:15:59.412749 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:15:59.412721    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-010013/test-model-010013-kserve-6bdfbb6f89-9fqsj"
Oct 30 05:16:00.374437 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:00.374408    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:01.666845 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:01.666803    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:01.780371 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:01.780332    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:02.199160 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:02.199122    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:03.439565 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:03.439529    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:03.535730 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:03.535699    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:04.462957 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:04.462924    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:06.133474 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:06.133439    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:07.869199 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:07.869166    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:08.232134 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:08.232042    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:16:09.665277 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:09.665229    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:11.568123 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:11.568090    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:13.312339 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:13.312238    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:13.709580 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:13.709550    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:13.809927 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:13.809876    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:14.997103 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:14.997053    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:15.754098 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:15.754052    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-8afb26/test-model-8afb26-kserve-6c6f6d8777-7mlmd"]
Oct 30 05:16:15.760429 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:15.760405    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8afb26/test-model-8afb26-kserve-6c6f6d8777-7mlmd"
Oct 30 05:16:15.773426 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:15.773398    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-8afb26/test-model-8afb26-kserve-6c6f6d8777-7mlmd"]
Oct 30 05:16:15.779596 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:15.779543    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-8afb26\"/\"default-dockercfg-cjtss\""
Oct 30 05:16:15.779596 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:15.779550    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-8afb26\"/\"test-model-8afb26-kserve-self-signed-certs\""
Oct 30 05:16:15.787702 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:15.787676    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:15.800808 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:15.800782    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-8afb26\"/\"openshift-service-ca.crt\""
Oct 30 05:16:15.800936 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:15.800867    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-8afb26\"/\"kube-root-ca.crt\""
Oct 30 05:16:15.883329 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:15.883297    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fac1e366-ef3a-45e5-bd96-8ebf290f0b5f-model-cache\") pod \"test-model-8afb26-kserve-6c6f6d8777-7mlmd\" (UID: \"fac1e366-ef3a-45e5-bd96-8ebf290f0b5f\") " pod="ai-tenant-e2e-inf-b-8afb26/test-model-8afb26-kserve-6c6f6d8777-7mlmd"
Oct 30 05:16:15.883329 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:15.883334    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fac1e366-ef3a-45e5-bd96-8ebf290f0b5f-dshm\") pod \"test-model-8afb26-kserve-6c6f6d8777-7mlmd\" (UID: \"fac1e366-ef3a-45e5-bd96-8ebf290f0b5f\") " pod="ai-tenant-e2e-inf-b-8afb26/test-model-8afb26-kserve-6c6f6d8777-7mlmd"
Oct 30 05:16:15.883605 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:15.883454    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fac1e366-ef3a-45e5-bd96-8ebf290f0b5f-tmp-dir\") pod \"test-model-8afb26-kserve-6c6f6d8777-7mlmd\" (UID: \"fac1e366-ef3a-45e5-bd96-8ebf290f0b5f\") " pod="ai-tenant-e2e-inf-b-8afb26/test-model-8afb26-kserve-6c6f6d8777-7mlmd"
Oct 30 05:16:15.883605 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:15.883555    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fac1e366-ef3a-45e5-bd96-8ebf290f0b5f-home\") pod \"test-model-8afb26-kserve-6c6f6d8777-7mlmd\" (UID: \"fac1e366-ef3a-45e5-bd96-8ebf290f0b5f\") " pod="ai-tenant-e2e-inf-b-8afb26/test-model-8afb26-kserve-6c6f6d8777-7mlmd"
Oct 30 05:16:15.883721 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:15.883649    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x7dk\" (UniqueName: \"kubernetes.io/projected/fac1e366-ef3a-45e5-bd96-8ebf290f0b5f-kube-api-access-2x7dk\") pod \"test-model-8afb26-kserve-6c6f6d8777-7mlmd\" (UID: \"fac1e366-ef3a-45e5-bd96-8ebf290f0b5f\") " pod="ai-tenant-e2e-inf-b-8afb26/test-model-8afb26-kserve-6c6f6d8777-7mlmd"
Oct 30 05:16:15.883721 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:15.883701    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fac1e366-ef3a-45e5-bd96-8ebf290f0b5f-tls-certs\") pod \"test-model-8afb26-kserve-6c6f6d8777-7mlmd\" (UID: \"fac1e366-ef3a-45e5-bd96-8ebf290f0b5f\") " pod="ai-tenant-e2e-inf-b-8afb26/test-model-8afb26-kserve-6c6f6d8777-7mlmd"
Oct 30 05:16:15.991681 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:15.986081    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fac1e366-ef3a-45e5-bd96-8ebf290f0b5f-tmp-dir\") pod \"test-model-8afb26-kserve-6c6f6d8777-7mlmd\" (UID: \"fac1e366-ef3a-45e5-bd96-8ebf290f0b5f\") " pod="ai-tenant-e2e-inf-b-8afb26/test-model-8afb26-kserve-6c6f6d8777-7mlmd"
Oct 30 05:16:15.991681 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:15.986260    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fac1e366-ef3a-45e5-bd96-8ebf290f0b5f-home\") pod \"test-model-8afb26-kserve-6c6f6d8777-7mlmd\" (UID: \"fac1e366-ef3a-45e5-bd96-8ebf290f0b5f\") " pod="ai-tenant-e2e-inf-b-8afb26/test-model-8afb26-kserve-6c6f6d8777-7mlmd"
Oct 30 05:16:15.991681 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:15.986320    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x7dk\" (UniqueName: \"kubernetes.io/projected/fac1e366-ef3a-45e5-bd96-8ebf290f0b5f-kube-api-access-2x7dk\") pod \"test-model-8afb26-kserve-6c6f6d8777-7mlmd\" (UID: \"fac1e366-ef3a-45e5-bd96-8ebf290f0b5f\") " pod="ai-tenant-e2e-inf-b-8afb26/test-model-8afb26-kserve-6c6f6d8777-7mlmd"
Oct 30 05:16:15.991681 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:15.986373    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fac1e366-ef3a-45e5-bd96-8ebf290f0b5f-tls-certs\") pod \"test-model-8afb26-kserve-6c6f6d8777-7mlmd\" (UID: \"fac1e366-ef3a-45e5-bd96-8ebf290f0b5f\") " pod="ai-tenant-e2e-inf-b-8afb26/test-model-8afb26-kserve-6c6f6d8777-7mlmd"
Oct 30 05:16:15.991681 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:15.986600    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fac1e366-ef3a-45e5-bd96-8ebf290f0b5f-model-cache\") pod \"test-model-8afb26-kserve-6c6f6d8777-7mlmd\" (UID: \"fac1e366-ef3a-45e5-bd96-8ebf290f0b5f\") " pod="ai-tenant-e2e-inf-b-8afb26/test-model-8afb26-kserve-6c6f6d8777-7mlmd"
Oct 30 05:16:15.991681 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:15.986632    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fac1e366-ef3a-45e5-bd96-8ebf290f0b5f-dshm\") pod \"test-model-8afb26-kserve-6c6f6d8777-7mlmd\" (UID: \"fac1e366-ef3a-45e5-bd96-8ebf290f0b5f\") " pod="ai-tenant-e2e-inf-b-8afb26/test-model-8afb26-kserve-6c6f6d8777-7mlmd"
Oct 30 05:16:15.991681 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:15.987728    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fac1e366-ef3a-45e5-bd96-8ebf290f0b5f-tmp-dir\") pod \"test-model-8afb26-kserve-6c6f6d8777-7mlmd\" (UID: \"fac1e366-ef3a-45e5-bd96-8ebf290f0b5f\") " pod="ai-tenant-e2e-inf-b-8afb26/test-model-8afb26-kserve-6c6f6d8777-7mlmd"
Oct 30 05:16:15.991681 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:15.988585    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fac1e366-ef3a-45e5-bd96-8ebf290f0b5f-home\") pod \"test-model-8afb26-kserve-6c6f6d8777-7mlmd\" (UID: \"fac1e366-ef3a-45e5-bd96-8ebf290f0b5f\") " pod="ai-tenant-e2e-inf-b-8afb26/test-model-8afb26-kserve-6c6f6d8777-7mlmd"
Oct 30 05:16:16.008967 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:15.993734    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fac1e366-ef3a-45e5-bd96-8ebf290f0b5f-model-cache\") pod \"test-model-8afb26-kserve-6c6f6d8777-7mlmd\" (UID: \"fac1e366-ef3a-45e5-bd96-8ebf290f0b5f\") " pod="ai-tenant-e2e-inf-b-8afb26/test-model-8afb26-kserve-6c6f6d8777-7mlmd"
Oct 30 05:16:16.008967 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:16.005539    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fac1e366-ef3a-45e5-bd96-8ebf290f0b5f-dshm\") pod \"test-model-8afb26-kserve-6c6f6d8777-7mlmd\" (UID: \"fac1e366-ef3a-45e5-bd96-8ebf290f0b5f\") " pod="ai-tenant-e2e-inf-b-8afb26/test-model-8afb26-kserve-6c6f6d8777-7mlmd"
Oct 30 05:16:16.015831 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:16.013617    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x7dk\" (UniqueName: \"kubernetes.io/projected/fac1e366-ef3a-45e5-bd96-8ebf290f0b5f-kube-api-access-2x7dk\") pod \"test-model-8afb26-kserve-6c6f6d8777-7mlmd\" (UID: \"fac1e366-ef3a-45e5-bd96-8ebf290f0b5f\") " pod="ai-tenant-e2e-inf-b-8afb26/test-model-8afb26-kserve-6c6f6d8777-7mlmd"
Oct 30 05:16:16.015831 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:16.014183    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fac1e366-ef3a-45e5-bd96-8ebf290f0b5f-tls-certs\") pod \"test-model-8afb26-kserve-6c6f6d8777-7mlmd\" (UID: \"fac1e366-ef3a-45e5-bd96-8ebf290f0b5f\") " pod="ai-tenant-e2e-inf-b-8afb26/test-model-8afb26-kserve-6c6f6d8777-7mlmd"
Oct 30 05:16:16.092672 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:16.092639    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8afb26/test-model-8afb26-kserve-6c6f6d8777-7mlmd"
Oct 30 05:16:16.450186 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:16.450140    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-8afb26/test-model-8afb26-kserve-6c6f6d8777-7mlmd"]
Oct 30 05:16:16.453019 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:16:16.452984    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfac1e366_ef3a_45e5_bd96_8ebf290f0b5f.slice/crio-8736229175be6e7d0344843dd6f5ee89acde16d742e2a2d4fa50c52c0a012868 WatchSource:0}: Error finding container 8736229175be6e7d0344843dd6f5ee89acde16d742e2a2d4fa50c52c0a012868: Status 404 returned error can't find the container with id 8736229175be6e7d0344843dd6f5ee89acde16d742e2a2d4fa50c52c0a012868
Oct 30 05:16:16.531635 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:16.531602    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8afb26/test-model-8afb26-kserve-6c6f6d8777-7mlmd" event={"ID":"fac1e366-ef3a-45e5-bd96-8ebf290f0b5f","Type":"ContainerStarted","Data":"8736229175be6e7d0344843dd6f5ee89acde16d742e2a2d4fa50c52c0a012868"}
Oct 30 05:16:16.675894 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:16.675863    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:17.540262 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:17.540223    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8afb26/test-model-8afb26-kserve-6c6f6d8777-7mlmd" event={"ID":"fac1e366-ef3a-45e5-bd96-8ebf290f0b5f","Type":"ContainerStarted","Data":"9cca65e2ccf236be9c73420b9d502eb88c8c55925e71e785df0db6fc69c53e9a"}
Oct 30 05:16:17.578729 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:17.578681    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-8afb26/test-model-8afb26-kserve-6c6f6d8777-7mlmd" podStartSLOduration=2.578669276 podStartE2EDuration="2.578669276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:16:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:16:17.578269758 +0000 UTC m=+1906.802089660" watchObservedRunningTime="2026-10-30 05:16:17.578669276 +0000 UTC m=+1906.802489194"
Oct 30 05:16:18.093122 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:18.093090    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-8afb26/test-model-8afb26-kserve-6c6f6d8777-7mlmd"
Oct 30 05:16:18.430499 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:18.430419    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:20.129865 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:20.129827    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:21.531139 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:21.531105    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:22.230745 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:22.230709    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:22.332796 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:22.332760    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:22.529615 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:22.529528    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:22.932109 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:22.932074    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:24.593129 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:24.593097    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:24.687448 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:24.687415    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:24.791242 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:24.791204    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:26.093805 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:26.093773    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-8afb26/test-model-8afb26-kserve-6c6f6d8777-7mlmd"
Oct 30 05:16:26.106099 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:26.106075    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-8afb26/test-model-8afb26-kserve-6c6f6d8777-7mlmd"
Oct 30 05:16:26.519278 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:26.519006    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:26.637364 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:26.637336    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-8afb26/test-model-8afb26-kserve-6c6f6d8777-7mlmd"
Oct 30 05:16:26.819316 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:16:26.819232    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:16:28.206682 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:28.206647    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:29.524369 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:29.524337    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:29.642873 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:29.642843    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:29.968396 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:29.968361    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:30.108926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:30.092862    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:30.806087 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:30.806053    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:31.325041 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:31.324996    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:31.435652 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:31.435616    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:32.160613 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:32.160578    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:32.560663 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:32.560587    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-mx96q"]
Oct 30 05:16:32.565115 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:32.565089    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-mx96q"
Oct 30 05:16:32.596589 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:32.596554    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-010013-openshift-default-dockercfg-z64dg\""
Oct 30 05:16:32.607938 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:32.607422    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-mx96q"]
Oct 30 05:16:32.638351 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:32.638310    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb90d6b7-a751-48d5-8d58-98c021a69145-istio-podinfo\") pod \"e2e-inf-a-010013-openshift-default-59977f9669-mx96q\" (UID: \"bb90d6b7-a751-48d5-8d58-98c021a69145\") " pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-mx96q"
Oct 30 05:16:32.638564 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:32.638375    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb90d6b7-a751-48d5-8d58-98c021a69145-istio-envoy\") pod \"e2e-inf-a-010013-openshift-default-59977f9669-mx96q\" (UID: \"bb90d6b7-a751-48d5-8d58-98c021a69145\") " pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-mx96q"
Oct 30 05:16:32.638564 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:32.638413    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb90d6b7-a751-48d5-8d58-98c021a69145-credential-socket\") pod \"e2e-inf-a-010013-openshift-default-59977f9669-mx96q\" (UID: \"bb90d6b7-a751-48d5-8d58-98c021a69145\") " pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-mx96q"
Oct 30 05:16:32.638564 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:32.638496    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb90d6b7-a751-48d5-8d58-98c021a69145-istio-token\") pod \"e2e-inf-a-010013-openshift-default-59977f9669-mx96q\" (UID: \"bb90d6b7-a751-48d5-8d58-98c021a69145\") " pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-mx96q"
Oct 30 05:16:32.638564 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:32.638524    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb90d6b7-a751-48d5-8d58-98c021a69145-workload-socket\") pod \"e2e-inf-a-010013-openshift-default-59977f9669-mx96q\" (UID: \"bb90d6b7-a751-48d5-8d58-98c021a69145\") " pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-mx96q"
Oct 30 05:16:32.638815 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:32.638626    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb90d6b7-a751-48d5-8d58-98c021a69145-istio-data\") pod \"e2e-inf-a-010013-openshift-default-59977f9669-mx96q\" (UID: \"bb90d6b7-a751-48d5-8d58-98c021a69145\") " pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-mx96q"
Oct 30 05:16:32.638815 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:32.638687    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb90d6b7-a751-48d5-8d58-98c021a69145-istiod-ca-cert\") pod \"e2e-inf-a-010013-openshift-default-59977f9669-mx96q\" (UID: \"bb90d6b7-a751-48d5-8d58-98c021a69145\") " pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-mx96q"
Oct 30 05:16:32.638815 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:32.638766    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dk64\" (UniqueName: \"kubernetes.io/projected/bb90d6b7-a751-48d5-8d58-98c021a69145-kube-api-access-8dk64\") pod \"e2e-inf-a-010013-openshift-default-59977f9669-mx96q\" (UID: \"bb90d6b7-a751-48d5-8d58-98c021a69145\") " pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-mx96q"
Oct 30 05:16:32.638815 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:32.638794    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb90d6b7-a751-48d5-8d58-98c021a69145-workload-certs\") pod \"e2e-inf-a-010013-openshift-default-59977f9669-mx96q\" (UID: \"bb90d6b7-a751-48d5-8d58-98c021a69145\") " pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-mx96q"
Oct 30 05:16:32.739722 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:32.739683    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb90d6b7-a751-48d5-8d58-98c021a69145-istio-data\") pod \"e2e-inf-a-010013-openshift-default-59977f9669-mx96q\" (UID: \"bb90d6b7-a751-48d5-8d58-98c021a69145\") " pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-mx96q"
Oct 30 05:16:32.739938 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:32.739752    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb90d6b7-a751-48d5-8d58-98c021a69145-istiod-ca-cert\") pod \"e2e-inf-a-010013-openshift-default-59977f9669-mx96q\" (UID: \"bb90d6b7-a751-48d5-8d58-98c021a69145\") " pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-mx96q"
Oct 30 05:16:32.739938 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:32.739819    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dk64\" (UniqueName: \"kubernetes.io/projected/bb90d6b7-a751-48d5-8d58-98c021a69145-kube-api-access-8dk64\") pod \"e2e-inf-a-010013-openshift-default-59977f9669-mx96q\" (UID: \"bb90d6b7-a751-48d5-8d58-98c021a69145\") " pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-mx96q"
Oct 30 05:16:32.739938 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:32.739845    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb90d6b7-a751-48d5-8d58-98c021a69145-workload-certs\") pod \"e2e-inf-a-010013-openshift-default-59977f9669-mx96q\" (UID: \"bb90d6b7-a751-48d5-8d58-98c021a69145\") " pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-mx96q"
Oct 30 05:16:32.740132 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:32.739978    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb90d6b7-a751-48d5-8d58-98c021a69145-istio-podinfo\") pod \"e2e-inf-a-010013-openshift-default-59977f9669-mx96q\" (UID: \"bb90d6b7-a751-48d5-8d58-98c021a69145\") " pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-mx96q"
Oct 30 05:16:32.740132 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:32.740019    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb90d6b7-a751-48d5-8d58-98c021a69145-istio-envoy\") pod \"e2e-inf-a-010013-openshift-default-59977f9669-mx96q\" (UID: \"bb90d6b7-a751-48d5-8d58-98c021a69145\") " pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-mx96q"
Oct 30 05:16:32.740132 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:32.740051    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb90d6b7-a751-48d5-8d58-98c021a69145-credential-socket\") pod \"e2e-inf-a-010013-openshift-default-59977f9669-mx96q\" (UID: \"bb90d6b7-a751-48d5-8d58-98c021a69145\") " pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-mx96q"
Oct 30 05:16:32.740294 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:32.740133    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb90d6b7-a751-48d5-8d58-98c021a69145-istio-token\") pod \"e2e-inf-a-010013-openshift-default-59977f9669-mx96q\" (UID: \"bb90d6b7-a751-48d5-8d58-98c021a69145\") " pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-mx96q"
Oct 30 05:16:32.740294 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:32.740165    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb90d6b7-a751-48d5-8d58-98c021a69145-workload-socket\") pod \"e2e-inf-a-010013-openshift-default-59977f9669-mx96q\" (UID: \"bb90d6b7-a751-48d5-8d58-98c021a69145\") " pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-mx96q"
Oct 30 05:16:32.740294 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:32.740193    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb90d6b7-a751-48d5-8d58-98c021a69145-istio-data\") pod \"e2e-inf-a-010013-openshift-default-59977f9669-mx96q\" (UID: \"bb90d6b7-a751-48d5-8d58-98c021a69145\") " pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-mx96q"
Oct 30 05:16:32.740294 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:32.740230    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb90d6b7-a751-48d5-8d58-98c021a69145-workload-certs\") pod \"e2e-inf-a-010013-openshift-default-59977f9669-mx96q\" (UID: \"bb90d6b7-a751-48d5-8d58-98c021a69145\") " pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-mx96q"
Oct 30 05:16:32.740523 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:32.740485    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb90d6b7-a751-48d5-8d58-98c021a69145-workload-socket\") pod \"e2e-inf-a-010013-openshift-default-59977f9669-mx96q\" (UID: \"bb90d6b7-a751-48d5-8d58-98c021a69145\") " pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-mx96q"
Oct 30 05:16:32.740587 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:32.740549    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb90d6b7-a751-48d5-8d58-98c021a69145-credential-socket\") pod \"e2e-inf-a-010013-openshift-default-59977f9669-mx96q\" (UID: \"bb90d6b7-a751-48d5-8d58-98c021a69145\") " pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-mx96q"
Oct 30 05:16:32.740684 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:32.740665    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb90d6b7-a751-48d5-8d58-98c021a69145-istiod-ca-cert\") pod \"e2e-inf-a-010013-openshift-default-59977f9669-mx96q\" (UID: \"bb90d6b7-a751-48d5-8d58-98c021a69145\") " pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-mx96q"
Oct 30 05:16:32.742916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:32.742881    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb90d6b7-a751-48d5-8d58-98c021a69145-istio-envoy\") pod \"e2e-inf-a-010013-openshift-default-59977f9669-mx96q\" (UID: \"bb90d6b7-a751-48d5-8d58-98c021a69145\") " pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-mx96q"
Oct 30 05:16:32.743131 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:32.743111    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb90d6b7-a751-48d5-8d58-98c021a69145-istio-podinfo\") pod \"e2e-inf-a-010013-openshift-default-59977f9669-mx96q\" (UID: \"bb90d6b7-a751-48d5-8d58-98c021a69145\") " pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-mx96q"
Oct 30 05:16:32.760631 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:32.760605    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb90d6b7-a751-48d5-8d58-98c021a69145-istio-token\") pod \"e2e-inf-a-010013-openshift-default-59977f9669-mx96q\" (UID: \"bb90d6b7-a751-48d5-8d58-98c021a69145\") " pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-mx96q"
Oct 30 05:16:32.760774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:32.760703    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dk64\" (UniqueName: \"kubernetes.io/projected/bb90d6b7-a751-48d5-8d58-98c021a69145-kube-api-access-8dk64\") pod \"e2e-inf-a-010013-openshift-default-59977f9669-mx96q\" (UID: \"bb90d6b7-a751-48d5-8d58-98c021a69145\") " pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-mx96q"
Oct 30 05:16:32.894850 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:32.894820    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-mx96q"
Oct 30 05:16:33.219380 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:33.219345    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-mx96q"]
Oct 30 05:16:33.220596 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:16:33.220560    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb90d6b7_a751_48d5_8d58_98c021a69145.slice/crio-fdc475c8108045f5a5303665bfad0eaddbf6bb2d0be4ce09fb20e0d5e8adb09c WatchSource:0}: Error finding container fdc475c8108045f5a5303665bfad0eaddbf6bb2d0be4ce09fb20e0d5e8adb09c: Status 404 returned error can't find the container with id fdc475c8108045f5a5303665bfad0eaddbf6bb2d0be4ce09fb20e0d5e8adb09c
Oct 30 05:16:33.223676 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:33.223639    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:16:33.223791 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:33.223707    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:16:33.223791 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:33.223735    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:16:33.700866 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:33.700823    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-mx96q" event={"ID":"bb90d6b7-a751-48d5-8d58-98c021a69145","Type":"ContainerStarted","Data":"c0c1099f6596b8fbbd6e3c33e9ecd44a017e3fe8c148977b397668e84a7ff086"}
Oct 30 05:16:33.701083 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:33.700892    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-mx96q" event={"ID":"bb90d6b7-a751-48d5-8d58-98c021a69145","Type":"ContainerStarted","Data":"fdc475c8108045f5a5303665bfad0eaddbf6bb2d0be4ce09fb20e0d5e8adb09c"}
Oct 30 05:16:33.758272 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:33.758213    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-mx96q" podStartSLOduration=1.7581953559999999 podStartE2EDuration="1.758195356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:16:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:16:33.75447491 +0000 UTC m=+1922.978294810" watchObservedRunningTime="2026-10-30 05:16:33.758195356 +0000 UTC m=+1922.982015258"
Oct 30 05:16:33.899106 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:33.899016    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-mx96q"
Oct 30 05:16:34.048258 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:34.048129    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:34.899014 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:34.898940    5898 patch_prober.go:29] interesting pod/e2e-inf-a-010013-openshift-default-59977f9669-mx96q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 05:16:34.899014 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:34.899005    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-mx96q" podUID="bb90d6b7-a751-48d5-8d58-98c021a69145" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 05:16:34.916772 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:34.916736    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:35.034641 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:35.034608    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:35.899287 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:35.899241    5898 patch_prober.go:29] interesting pod/e2e-inf-a-010013-openshift-default-59977f9669-mx96q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 05:16:35.899769 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:35.899299    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-mx96q" podUID="bb90d6b7-a751-48d5-8d58-98c021a69145" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 05:16:36.101420 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:36.101373    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:36.672252 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:36.672223    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:36.798959 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:36.798356    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:36.902647 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:36.902601    5898 patch_prober.go:29] interesting pod/e2e-inf-a-010013-openshift-default-59977f9669-mx96q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 05:16:36.903302 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:36.903265    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-mx96q" podUID="bb90d6b7-a751-48d5-8d58-98c021a69145" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 05:16:37.898923 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:37.898225    5898 patch_prober.go:29] interesting pod/e2e-inf-a-010013-openshift-default-59977f9669-mx96q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 05:16:37.898923 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:37.898296    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-mx96q" podUID="bb90d6b7-a751-48d5-8d58-98c021a69145" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 05:16:38.090928 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:38.086379    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:38.897354 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:38.897317    5898 patch_prober.go:29] interesting pod/e2e-inf-a-010013-openshift-default-59977f9669-mx96q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 05:16:38.899146 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:38.899110    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-mx96q" podUID="bb90d6b7-a751-48d5-8d58-98c021a69145" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 05:16:39.763180 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:39.763146    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:40.002963 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:40.002934    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-mx96q"
Oct 30 05:16:40.004121 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:40.004067    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-mx96q"
Oct 30 05:16:40.006464 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:40.006438    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-mx96q"
Oct 30 05:16:41.211449 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:41.211417    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:41.514501 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:41.514417    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:43.441447 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:43.441415    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:44.804586 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:44.804548    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:44.901409 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:44.901359    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:45.246706 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:45.246677    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:45.384120 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:45.384087    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:45.485172 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:45.482204    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:45.871533 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:45.871488    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:46.406162 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:46.406127    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:46.913418 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:46.913390    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:47.070324 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:47.070290    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:47.646652 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:47.646620    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:49.617620 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:49.617585    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:50.457316 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:50.457281    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:50.565191 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:50.565162    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:50.665733 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:50.665676    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:51.553015 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:51.552980    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:52.183135 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:52.183103    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:52.278553 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:52.278522    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:53.425083 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:53.425050    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:55.085560 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:55.085529    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:56.816630 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:56.816600    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:16:56.819965 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:16:56.819935    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:16:58.602922 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:16:58.600614    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:17:00.519073 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:00.519034    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:17:02.300929 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:02.300557    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:17:04.131279 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:04.131232    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:17:06.068892 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:06.068817    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:17:07.755093 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:07.755061    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:17:08.608487 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:08.608451    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:17:09.386772 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:09.386720    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:17:09.982629 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:09.982595    5898 generic.go:356] "Generic (PLEG): container finished" podID="e1d94e3b-c651-4989-afbe-53acbd131f71" containerID="d56328b64cfc56d88c83488b4f04061781785f631a042f33f74b30d94e1dae95" exitCode=0
Oct 30 05:17:09.982811 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:09.982670    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p" event={"ID":"e1d94e3b-c651-4989-afbe-53acbd131f71","Type":"ContainerDied","Data":"d56328b64cfc56d88c83488b4f04061781785f631a042f33f74b30d94e1dae95"}
Oct 30 05:17:09.983123 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:09.983106    5898 scope.go:122] "RemoveContainer" containerID="d56328b64cfc56d88c83488b4f04061781785f631a042f33f74b30d94e1dae95"
Oct 30 05:17:09.984893 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:09.984866    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:17:09.984986 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:09.984952    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:17:09.985029 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:09.984990    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:17:10.235050 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:10.234965    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p"
Oct 30 05:17:10.722301 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:10.722263    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9"]
Oct 30 05:17:10.722798 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:10.722464    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9" podUID="d62d9b95-5024-49d1-88f4-048477578a45" containerName="istio-proxy" containerID="cri-o://42bf18f22b3abc85d7391e970ec6c296716b394e068e8ec33a998416b51b9f5b" gracePeriod=30
Oct 30 05:17:11.006828 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:11.006734    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p" event={"ID":"e1d94e3b-c651-4989-afbe-53acbd131f71","Type":"ContainerStarted","Data":"30a82c2d3d8a70b02954e5140ecd779b5f9f043f8aba447e23006e3d57d7f3c3"}
Oct 30 05:17:11.265247 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:11.265157    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:17:11.931623 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:11.931588    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p"
Oct 30 05:17:11.932613 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:11.932592    5898 patch_prober.go:29] interesting pod/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Oct 30 05:17:11.932678 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:11.932647    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p" podUID="e1d94e3b-c651-4989-afbe-53acbd131f71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Oct 30 05:17:12.933006 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:12.932970    5898 patch_prober.go:29] interesting pod/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Oct 30 05:17:12.933684 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:12.933651    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p" podUID="e1d94e3b-c651-4989-afbe-53acbd131f71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Oct 30 05:17:12.946516 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:12.946491    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:17:13.900460 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:13.900427    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:17:13.931371 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:13.931340    5898 patch_prober.go:29] interesting pod/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Oct 30 05:17:13.931521 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:13.931391    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p" podUID="e1d94e3b-c651-4989-afbe-53acbd131f71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Oct 30 05:17:14.899779 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:14.898322    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:17:14.932339 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:14.932306    5898 patch_prober.go:29] interesting pod/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Oct 30 05:17:14.932534 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:14.932363    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p" podUID="e1d94e3b-c651-4989-afbe-53acbd131f71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Oct 30 05:17:15.672694 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:15.672662    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:17:15.935921 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:15.935323    5898 patch_prober.go:29] interesting pod/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Oct 30 05:17:15.935921 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:15.935400    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p" podUID="e1d94e3b-c651-4989-afbe-53acbd131f71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Oct 30 05:17:16.168841 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:16.168800    5898 generic.go:356] "Generic (PLEG): container finished" podID="d62d9b95-5024-49d1-88f4-048477578a45" containerID="42bf18f22b3abc85d7391e970ec6c296716b394e068e8ec33a998416b51b9f5b" exitCode=0
Oct 30 05:17:16.169169 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:16.169142    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9" event={"ID":"d62d9b95-5024-49d1-88f4-048477578a45","Type":"ContainerDied","Data":"42bf18f22b3abc85d7391e970ec6c296716b394e068e8ec33a998416b51b9f5b"}
Oct 30 05:17:16.259641 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:16.259614    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9"
Oct 30 05:17:16.430345 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:16.430303    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d62d9b95-5024-49d1-88f4-048477578a45-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d62d9b95-5024-49d1-88f4-048477578a45-istio-data\") pod \"d62d9b95-5024-49d1-88f4-048477578a45\" (UID: \"d62d9b95-5024-49d1-88f4-048477578a45\") "
Oct 30 05:17:16.430759 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:16.430695    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d62d9b95-5024-49d1-88f4-048477578a45-istio-data" pod "d62d9b95-5024-49d1-88f4-048477578a45" (UID: "d62d9b95-5024-49d1-88f4-048477578a45"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:17:16.430759 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:16.430706    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d62d9b95-5024-49d1-88f4-048477578a45-kube-api-access-zz29j\" (UniqueName: \"kubernetes.io/projected/d62d9b95-5024-49d1-88f4-048477578a45-kube-api-access-zz29j\") pod \"d62d9b95-5024-49d1-88f4-048477578a45\" (UID: \"d62d9b95-5024-49d1-88f4-048477578a45\") "
Oct 30 05:17:16.430966 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:16.430798    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d62d9b95-5024-49d1-88f4-048477578a45-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62d9b95-5024-49d1-88f4-048477578a45-workload-socket\") pod \"d62d9b95-5024-49d1-88f4-048477578a45\" (UID: \"d62d9b95-5024-49d1-88f4-048477578a45\") "
Oct 30 05:17:16.430966 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:16.430885    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d62d9b95-5024-49d1-88f4-048477578a45-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62d9b95-5024-49d1-88f4-048477578a45-credential-socket\") pod \"d62d9b95-5024-49d1-88f4-048477578a45\" (UID: \"d62d9b95-5024-49d1-88f4-048477578a45\") "
Oct 30 05:17:16.431092 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:16.430963    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d62d9b95-5024-49d1-88f4-048477578a45-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d62d9b95-5024-49d1-88f4-048477578a45-istio-envoy\") pod \"d62d9b95-5024-49d1-88f4-048477578a45\" (UID: \"d62d9b95-5024-49d1-88f4-048477578a45\") "
Oct 30 05:17:16.431347 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:16.431110    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d62d9b95-5024-49d1-88f4-048477578a45-workload-socket" pod "d62d9b95-5024-49d1-88f4-048477578a45" (UID: "d62d9b95-5024-49d1-88f4-048477578a45"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:17:16.431475 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:16.431146    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d62d9b95-5024-49d1-88f4-048477578a45-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d62d9b95-5024-49d1-88f4-048477578a45-istiod-ca-cert\") pod \"d62d9b95-5024-49d1-88f4-048477578a45\" (UID: \"d62d9b95-5024-49d1-88f4-048477578a45\") "
Oct 30 05:17:16.431475 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:16.431251    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d62d9b95-5024-49d1-88f4-048477578a45-credential-socket" pod "d62d9b95-5024-49d1-88f4-048477578a45" (UID: "d62d9b95-5024-49d1-88f4-048477578a45"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:17:16.431673 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:16.431498    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d62d9b95-5024-49d1-88f4-048477578a45-istio-token\" (UniqueName: \"kubernetes.io/projected/d62d9b95-5024-49d1-88f4-048477578a45-istio-token\") pod \"d62d9b95-5024-49d1-88f4-048477578a45\" (UID: \"d62d9b95-5024-49d1-88f4-048477578a45\") "
Oct 30 05:17:16.431673 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:16.431547    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d62d9b95-5024-49d1-88f4-048477578a45-istiod-ca-cert" pod "d62d9b95-5024-49d1-88f4-048477578a45" (UID: "d62d9b95-5024-49d1-88f4-048477578a45"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:17:16.431673 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:16.431562    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d62d9b95-5024-49d1-88f4-048477578a45-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d62d9b95-5024-49d1-88f4-048477578a45-istio-podinfo\") pod \"d62d9b95-5024-49d1-88f4-048477578a45\" (UID: \"d62d9b95-5024-49d1-88f4-048477578a45\") "
Oct 30 05:17:16.431673 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:16.431619    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d62d9b95-5024-49d1-88f4-048477578a45-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d62d9b95-5024-49d1-88f4-048477578a45-workload-certs\") pod \"d62d9b95-5024-49d1-88f4-048477578a45\" (UID: \"d62d9b95-5024-49d1-88f4-048477578a45\") "
Oct 30 05:17:16.432165 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:16.432141    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d62d9b95-5024-49d1-88f4-048477578a45-workload-certs" pod "d62d9b95-5024-49d1-88f4-048477578a45" (UID: "d62d9b95-5024-49d1-88f4-048477578a45"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:17:16.432752 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:16.432683    5898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d62d9b95-5024-49d1-88f4-048477578a45-istiod-ca-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:17:16.432752 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:16.432707    5898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d62d9b95-5024-49d1-88f4-048477578a45-workload-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:17:16.432752 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:16.432718    5898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d62d9b95-5024-49d1-88f4-048477578a45-istio-data\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:17:16.432752 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:16.432726    5898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62d9b95-5024-49d1-88f4-048477578a45-workload-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:17:16.432752 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:16.432734    5898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62d9b95-5024-49d1-88f4-048477578a45-credential-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:17:16.433551 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:16.433525    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d62d9b95-5024-49d1-88f4-048477578a45-kube-api-access-zz29j" pod "d62d9b95-5024-49d1-88f4-048477578a45" (UID: "d62d9b95-5024-49d1-88f4-048477578a45"). InnerVolumeSpecName "kube-api-access-zz29j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:17:16.434130 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:16.434109    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d62d9b95-5024-49d1-88f4-048477578a45-istio-podinfo" pod "d62d9b95-5024-49d1-88f4-048477578a45" (UID: "d62d9b95-5024-49d1-88f4-048477578a45"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:17:16.434212 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:16.434193    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d62d9b95-5024-49d1-88f4-048477578a45-istio-envoy" pod "d62d9b95-5024-49d1-88f4-048477578a45" (UID: "d62d9b95-5024-49d1-88f4-048477578a45"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:17:16.434291 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:16.434273    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d62d9b95-5024-49d1-88f4-048477578a45-istio-token" pod "d62d9b95-5024-49d1-88f4-048477578a45" (UID: "d62d9b95-5024-49d1-88f4-048477578a45"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:17:16.534098 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:16.534012    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz29j\" (UniqueName: \"kubernetes.io/projected/d62d9b95-5024-49d1-88f4-048477578a45-kube-api-access-zz29j\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:17:16.534098 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:16.534041    5898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d62d9b95-5024-49d1-88f4-048477578a45-istio-envoy\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:17:16.534098 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:16.534054    5898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d62d9b95-5024-49d1-88f4-048477578a45-istio-token\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:17:16.534098 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:16.534064    5898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d62d9b95-5024-49d1-88f4-048477578a45-istio-podinfo\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:17:16.560725 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:16.560690    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:17:16.936495 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:16.936467    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p"
Oct 30 05:17:16.936935 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:16.936922    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p"
Oct 30 05:17:16.937709 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:16.937690    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p"
Oct 30 05:17:17.184854 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:17.184772    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9"
Oct 30 05:17:17.184854 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:17.184792    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9" event={"ID":"d62d9b95-5024-49d1-88f4-048477578a45","Type":"ContainerDied","Data":"52b62a664410fb2af5b4d5c6ab67644912afb37acde83a0d4a8164f1038a5f6a"}
Oct 30 05:17:17.185110 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:17.184887    5898 scope.go:122] "RemoveContainer" containerID="42bf18f22b3abc85d7391e970ec6c296716b394e068e8ec33a998416b51b9f5b"
Oct 30 05:17:17.234844 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:17.234810    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9"]
Oct 30 05:17:17.240563 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:17.240535    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-d2pk9"]
Oct 30 05:17:18.250051 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:18.250022    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:17:18.894286 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:18.894253    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d62d9b95-5024-49d1-88f4-048477578a45" path="/var/lib/kubelet/pods/d62d9b95-5024-49d1-88f4-048477578a45/volumes"
Oct 30 05:17:19.790975 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:19.790946    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-802b8a-6bdbc888b-7645w"]
Oct 30 05:17:19.791566 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:19.791553    5898 cpu_manager.go:414] "containerMap: removing container" podUID="d62d9b95-5024-49d1-88f4-048477578a45" containerName="istio-proxy"
Oct 30 05:17:19.791628 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:19.791567    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d62d9b95-5024-49d1-88f4-048477578a45" containerName="istio-proxy"
Oct 30 05:17:19.791694 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:19.791684    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="d62d9b95-5024-49d1-88f4-048477578a45" containerName="istio-proxy"
Oct 30 05:17:19.794999 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:19.794980    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-802b8a-6bdbc888b-7645w"
Oct 30 05:17:19.811850 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:19.811827    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-802b8a-6bdbc888b-7645w"]
Oct 30 05:17:19.818607 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:19.818585    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-802b8a-serving-cert\""
Oct 30 05:17:19.873453 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:19.873420    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:17:19.889232 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:19.889203    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27c6e30a-940d-4fcf-b229-16a706b90709-metrics-certs\") pod \"maas-api-e2e-inf-a-802b8a-6bdbc888b-7645w\" (UID: \"27c6e30a-940d-4fcf-b229-16a706b90709\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-802b8a-6bdbc888b-7645w"
Oct 30 05:17:19.889407 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:19.889260    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27c6e30a-940d-4fcf-b229-16a706b90709-openshift-service-ca\") pod \"maas-api-e2e-inf-a-802b8a-6bdbc888b-7645w\" (UID: \"27c6e30a-940d-4fcf-b229-16a706b90709\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-802b8a-6bdbc888b-7645w"
Oct 30 05:17:19.889407 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:19.889296    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27c6e30a-940d-4fcf-b229-16a706b90709-maas-api-tls\") pod \"maas-api-e2e-inf-a-802b8a-6bdbc888b-7645w\" (UID: \"27c6e30a-940d-4fcf-b229-16a706b90709\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-802b8a-6bdbc888b-7645w"
Oct 30 05:17:19.889407 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:19.889399    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgbrf\" (UniqueName: \"kubernetes.io/projected/27c6e30a-940d-4fcf-b229-16a706b90709-kube-api-access-rgbrf\") pod \"maas-api-e2e-inf-a-802b8a-6bdbc888b-7645w\" (UID: \"27c6e30a-940d-4fcf-b229-16a706b90709\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-802b8a-6bdbc888b-7645w"
Oct 30 05:17:19.975367 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:19.971785    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:17:19.995491 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:19.990761    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgbrf\" (UniqueName: \"kubernetes.io/projected/27c6e30a-940d-4fcf-b229-16a706b90709-kube-api-access-rgbrf\") pod \"maas-api-e2e-inf-a-802b8a-6bdbc888b-7645w\" (UID: \"27c6e30a-940d-4fcf-b229-16a706b90709\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-802b8a-6bdbc888b-7645w"
Oct 30 05:17:19.995491 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:19.994435    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27c6e30a-940d-4fcf-b229-16a706b90709-metrics-certs\") pod \"maas-api-e2e-inf-a-802b8a-6bdbc888b-7645w\" (UID: \"27c6e30a-940d-4fcf-b229-16a706b90709\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-802b8a-6bdbc888b-7645w"
Oct 30 05:17:19.995491 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:19.994496    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27c6e30a-940d-4fcf-b229-16a706b90709-openshift-service-ca\") pod \"maas-api-e2e-inf-a-802b8a-6bdbc888b-7645w\" (UID: \"27c6e30a-940d-4fcf-b229-16a706b90709\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-802b8a-6bdbc888b-7645w"
Oct 30 05:17:19.995491 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:19.994567    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27c6e30a-940d-4fcf-b229-16a706b90709-maas-api-tls\") pod \"maas-api-e2e-inf-a-802b8a-6bdbc888b-7645w\" (UID: \"27c6e30a-940d-4fcf-b229-16a706b90709\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-802b8a-6bdbc888b-7645w"
Oct 30 05:17:19.997601 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:17:19.997575    5898 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-802b8a-serving-cert: secret "maas-api-e2e-inf-a-802b8a-serving-cert" not found
Oct 30 05:17:19.997743 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:17:19.997644    5898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27c6e30a-940d-4fcf-b229-16a706b90709-maas-api-tls podName:27c6e30a-940d-4fcf-b229-16a706b90709 nodeName:}" failed. No retries permitted until 2026-10-30 05:17:20.497624357 +0000 UTC m=+1969.721444238 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/27c6e30a-940d-4fcf-b229-16a706b90709-maas-api-tls") pod "maas-api-e2e-inf-a-802b8a-6bdbc888b-7645w" (UID: "27c6e30a-940d-4fcf-b229-16a706b90709") : secret "maas-api-e2e-inf-a-802b8a-serving-cert" not found
Oct 30 05:17:19.999118 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:19.999089    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27c6e30a-940d-4fcf-b229-16a706b90709-openshift-service-ca\") pod \"maas-api-e2e-inf-a-802b8a-6bdbc888b-7645w\" (UID: \"27c6e30a-940d-4fcf-b229-16a706b90709\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-802b8a-6bdbc888b-7645w"
Oct 30 05:17:20.003703 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:20.003670    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27c6e30a-940d-4fcf-b229-16a706b90709-metrics-certs\") pod \"maas-api-e2e-inf-a-802b8a-6bdbc888b-7645w\" (UID: \"27c6e30a-940d-4fcf-b229-16a706b90709\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-802b8a-6bdbc888b-7645w"
Oct 30 05:17:20.016575 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:20.015084    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgbrf\" (UniqueName: \"kubernetes.io/projected/27c6e30a-940d-4fcf-b229-16a706b90709-kube-api-access-rgbrf\") pod \"maas-api-e2e-inf-a-802b8a-6bdbc888b-7645w\" (UID: \"27c6e30a-940d-4fcf-b229-16a706b90709\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-802b8a-6bdbc888b-7645w"
Oct 30 05:17:20.207488 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:20.207454    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:17:20.502043 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:20.501949    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27c6e30a-940d-4fcf-b229-16a706b90709-maas-api-tls\") pod \"maas-api-e2e-inf-a-802b8a-6bdbc888b-7645w\" (UID: \"27c6e30a-940d-4fcf-b229-16a706b90709\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-802b8a-6bdbc888b-7645w"
Oct 30 05:17:20.504367 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:20.504343    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27c6e30a-940d-4fcf-b229-16a706b90709-maas-api-tls\") pod \"maas-api-e2e-inf-a-802b8a-6bdbc888b-7645w\" (UID: \"27c6e30a-940d-4fcf-b229-16a706b90709\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-802b8a-6bdbc888b-7645w"
Oct 30 05:17:20.708398 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:20.708364    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-802b8a-6bdbc888b-7645w"
Oct 30 05:17:21.018515 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:21.018485    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-802b8a-6bdbc888b-7645w"]
Oct 30 05:17:21.020173 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:17:21.020112    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27c6e30a_940d_4fcf_b229_16a706b90709.slice/crio-c84396b1c25ca8b085828be4f0417199c1a8000e15cb9baeb0585b7a7c2ad635 WatchSource:0}: Error finding container c84396b1c25ca8b085828be4f0417199c1a8000e15cb9baeb0585b7a7c2ad635: Status 404 returned error can't find the container with id c84396b1c25ca8b085828be4f0417199c1a8000e15cb9baeb0585b7a7c2ad635
Oct 30 05:17:21.217304 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:21.217268    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-802b8a-6bdbc888b-7645w" event={"ID":"27c6e30a-940d-4fcf-b229-16a706b90709","Type":"ContainerStarted","Data":"c84396b1c25ca8b085828be4f0417199c1a8000e15cb9baeb0585b7a7c2ad635"}
Oct 30 05:17:21.656794 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:21.656706    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:17:22.226629 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:22.226590    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-802b8a-6bdbc888b-7645w" event={"ID":"27c6e30a-940d-4fcf-b229-16a706b90709","Type":"ContainerStarted","Data":"318a9520eba0ed81aaca02474bd53b2a92fda138d1c4f132cdb0572f02606a40"}
Oct 30 05:17:22.227085 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:22.226699    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-802b8a-6bdbc888b-7645w"
Oct 30 05:17:22.266893 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:22.266719    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-802b8a-6bdbc888b-7645w" podStartSLOduration=2.941284669 podStartE2EDuration="3.266700901s" totalImagesPullingTime="325.416232ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:17:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:17:22.265083229 +0000 UTC m=+1971.488903159" watchObservedRunningTime="2026-10-30 05:17:22.266700901 +0000 UTC m=+1971.490520801"
Oct 30 05:17:23.402285 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:23.402253    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:17:25.124654 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:25.124622    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:17:26.818484 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:17:26.818445    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:17:26.950838 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:26.950802    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:17:28.239088 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:28.239058    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-802b8a-6bdbc888b-7645w"
Oct 30 05:17:28.378044 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:28.378008    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:17:28.786522 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:28.786486    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:17:30.806660 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:30.806622    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:17:32.535081 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:32.535053    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:17:34.479476 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:34.479441    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:17:36.340100 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:36.340069    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-a914e9-777dc7878c-j4drn"]
Oct 30 05:17:36.345615 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:36.345594    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-a914e9-777dc7878c-j4drn"
Oct 30 05:17:36.363884 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:36.363856    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-a914e9-777dc7878c-j4drn"]
Oct 30 05:17:36.370465 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:36.369978    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w6-a914e9-serving-cert\""
Oct 30 05:17:36.440244 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:36.440212    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/132139a5-1000-4abb-8964-d721b3a1bc37-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-a914e9-777dc7878c-j4drn\" (UID: \"132139a5-1000-4abb-8964-d721b3a1bc37\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-a914e9-777dc7878c-j4drn"
Oct 30 05:17:36.440444 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:36.440278    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/132139a5-1000-4abb-8964-d721b3a1bc37-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-a914e9-777dc7878c-j4drn\" (UID: \"132139a5-1000-4abb-8964-d721b3a1bc37\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-a914e9-777dc7878c-j4drn"
Oct 30 05:17:36.440444 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:36.440368    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgq9d\" (UniqueName: \"kubernetes.io/projected/132139a5-1000-4abb-8964-d721b3a1bc37-kube-api-access-tgq9d\") pod \"maas-api-e2e-shared-b-w6-a914e9-777dc7878c-j4drn\" (UID: \"132139a5-1000-4abb-8964-d721b3a1bc37\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-a914e9-777dc7878c-j4drn"
Oct 30 05:17:36.440444 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:36.440422    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/132139a5-1000-4abb-8964-d721b3a1bc37-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-a914e9-777dc7878c-j4drn\" (UID: \"132139a5-1000-4abb-8964-d721b3a1bc37\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-a914e9-777dc7878c-j4drn"
Oct 30 05:17:36.455702 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:36.455677    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:17:36.544852 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:36.544822    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/132139a5-1000-4abb-8964-d721b3a1bc37-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-a914e9-777dc7878c-j4drn\" (UID: \"132139a5-1000-4abb-8964-d721b3a1bc37\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-a914e9-777dc7878c-j4drn"
Oct 30 05:17:36.545112 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:36.544912    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/132139a5-1000-4abb-8964-d721b3a1bc37-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-a914e9-777dc7878c-j4drn\" (UID: \"132139a5-1000-4abb-8964-d721b3a1bc37\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-a914e9-777dc7878c-j4drn"
Oct 30 05:17:36.545112 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:36.544953    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgq9d\" (UniqueName: \"kubernetes.io/projected/132139a5-1000-4abb-8964-d721b3a1bc37-kube-api-access-tgq9d\") pod \"maas-api-e2e-shared-b-w6-a914e9-777dc7878c-j4drn\" (UID: \"132139a5-1000-4abb-8964-d721b3a1bc37\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-a914e9-777dc7878c-j4drn"
Oct 30 05:17:36.545112 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:36.544990    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/132139a5-1000-4abb-8964-d721b3a1bc37-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-a914e9-777dc7878c-j4drn\" (UID: \"132139a5-1000-4abb-8964-d721b3a1bc37\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-a914e9-777dc7878c-j4drn"
Oct 30 05:17:36.546095 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:36.546068    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/132139a5-1000-4abb-8964-d721b3a1bc37-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-a914e9-777dc7878c-j4drn\" (UID: \"132139a5-1000-4abb-8964-d721b3a1bc37\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-a914e9-777dc7878c-j4drn"
Oct 30 05:17:36.560931 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:36.560730    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/132139a5-1000-4abb-8964-d721b3a1bc37-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-a914e9-777dc7878c-j4drn\" (UID: \"132139a5-1000-4abb-8964-d721b3a1bc37\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-a914e9-777dc7878c-j4drn"
Oct 30 05:17:36.563369 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:36.563297    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/132139a5-1000-4abb-8964-d721b3a1bc37-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-a914e9-777dc7878c-j4drn\" (UID: \"132139a5-1000-4abb-8964-d721b3a1bc37\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-a914e9-777dc7878c-j4drn"
Oct 30 05:17:36.590825 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:36.587690    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgq9d\" (UniqueName: \"kubernetes.io/projected/132139a5-1000-4abb-8964-d721b3a1bc37-kube-api-access-tgq9d\") pod \"maas-api-e2e-shared-b-w6-a914e9-777dc7878c-j4drn\" (UID: \"132139a5-1000-4abb-8964-d721b3a1bc37\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-a914e9-777dc7878c-j4drn"
Oct 30 05:17:36.660881 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:36.660843    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-a914e9-777dc7878c-j4drn"
Oct 30 05:17:36.719333 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:36.719300    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:17:36.971632 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:36.971603    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-a914e9-777dc7878c-j4drn"]
Oct 30 05:17:36.972290 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:17:36.972264    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod132139a5_1000_4abb_8964_d721b3a1bc37.slice/crio-f7975768c3910fc94ced81bf7d4794f8aee0f7c15571f1be181a7f56fd9d8cb3 WatchSource:0}: Error finding container f7975768c3910fc94ced81bf7d4794f8aee0f7c15571f1be181a7f56fd9d8cb3: Status 404 returned error can't find the container with id f7975768c3910fc94ced81bf7d4794f8aee0f7c15571f1be181a7f56fd9d8cb3
Oct 30 05:17:37.392620 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:37.392580    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-a914e9-777dc7878c-j4drn" event={"ID":"132139a5-1000-4abb-8964-d721b3a1bc37","Type":"ContainerStarted","Data":"f7975768c3910fc94ced81bf7d4794f8aee0f7c15571f1be181a7f56fd9d8cb3"}
Oct 30 05:17:38.366658 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:38.366611    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:17:38.402616 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:38.402578    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-a914e9-777dc7878c-j4drn" event={"ID":"132139a5-1000-4abb-8964-d721b3a1bc37","Type":"ContainerStarted","Data":"cdc8942d9db2c701ccb2f4c339c37324bbc7949f0e4e8af925bf7b7b92af8ae9"}
Oct 30 05:17:38.403133 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:38.402649    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-a914e9-777dc7878c-j4drn"
Oct 30 05:17:38.457718 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:38.457662    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-a914e9-777dc7878c-j4drn" podStartSLOduration=2.09985222 podStartE2EDuration="2.457648526s" totalImagesPullingTime="357.796306ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:17:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:17:38.456634956 +0000 UTC m=+1987.680454878" watchObservedRunningTime="2026-10-30 05:17:38.457648526 +0000 UTC m=+1987.681468427"
Oct 30 05:17:39.097252 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:39.097219    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:17:40.167094 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:40.167063    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:17:41.331663 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:41.331626    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:17:42.098425 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:42.098351    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:17:43.801312 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:43.801283    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:17:44.417083 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:44.417054    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-a914e9-777dc7878c-j4drn"
Oct 30 05:17:45.451037 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:45.451006    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:17:47.113004 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:47.112966    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:17:47.922525 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:47.922490    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:17:48.787273 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:48.787230    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:17:50.411387 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:50.411336    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:17:52.292305 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:52.292276    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:17:54.034057 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:54.034027    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:17:54.623252 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:54.623211    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-356cae/test-model-356cae-kserve-78bbbb868f-bmnn8"]
Oct 30 05:17:54.628771 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:54.628742    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-356cae/test-model-356cae-kserve-78bbbb868f-bmnn8"
Oct 30 05:17:54.644770 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:54.644741    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-356cae/test-model-356cae-kserve-78bbbb868f-bmnn8"]
Oct 30 05:17:54.650954 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:54.650886    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-356cae\"/\"openshift-service-ca.crt\""
Oct 30 05:17:54.651074 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:54.650974    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-356cae\"/\"test-model-356cae-kserve-self-signed-certs\""
Oct 30 05:17:54.654960 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:54.654938    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-356cae\"/\"kube-root-ca.crt\""
Oct 30 05:17:54.655071 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:54.655008    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-356cae\"/\"default-dockercfg-f8n96\""
Oct 30 05:17:54.666579 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:54.666556    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:17:54.742753 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:54.742709    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee5dfade-6b22-4653-80d9-75320b4c6117-home\") pod \"test-model-356cae-kserve-78bbbb868f-bmnn8\" (UID: \"ee5dfade-6b22-4653-80d9-75320b4c6117\") " pod="ai-tenant-e2e-inf-a-356cae/test-model-356cae-kserve-78bbbb868f-bmnn8"
Oct 30 05:17:54.742971 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:54.742874    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee5dfade-6b22-4653-80d9-75320b4c6117-model-cache\") pod \"test-model-356cae-kserve-78bbbb868f-bmnn8\" (UID: \"ee5dfade-6b22-4653-80d9-75320b4c6117\") " pod="ai-tenant-e2e-inf-a-356cae/test-model-356cae-kserve-78bbbb868f-bmnn8"
Oct 30 05:17:54.743051 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:54.742981    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee5dfade-6b22-4653-80d9-75320b4c6117-tls-certs\") pod \"test-model-356cae-kserve-78bbbb868f-bmnn8\" (UID: \"ee5dfade-6b22-4653-80d9-75320b4c6117\") " pod="ai-tenant-e2e-inf-a-356cae/test-model-356cae-kserve-78bbbb868f-bmnn8"
Oct 30 05:17:54.743205 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:54.743181    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee5dfade-6b22-4653-80d9-75320b4c6117-tmp-dir\") pod \"test-model-356cae-kserve-78bbbb868f-bmnn8\" (UID: \"ee5dfade-6b22-4653-80d9-75320b4c6117\") " pod="ai-tenant-e2e-inf-a-356cae/test-model-356cae-kserve-78bbbb868f-bmnn8"
Oct 30 05:17:54.743499 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:54.743477    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68zvk\" (UniqueName: \"kubernetes.io/projected/ee5dfade-6b22-4653-80d9-75320b4c6117-kube-api-access-68zvk\") pod \"test-model-356cae-kserve-78bbbb868f-bmnn8\" (UID: \"ee5dfade-6b22-4653-80d9-75320b4c6117\") " pod="ai-tenant-e2e-inf-a-356cae/test-model-356cae-kserve-78bbbb868f-bmnn8"
Oct 30 05:17:54.743720 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:54.743698    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee5dfade-6b22-4653-80d9-75320b4c6117-dshm\") pod \"test-model-356cae-kserve-78bbbb868f-bmnn8\" (UID: \"ee5dfade-6b22-4653-80d9-75320b4c6117\") " pod="ai-tenant-e2e-inf-a-356cae/test-model-356cae-kserve-78bbbb868f-bmnn8"
Oct 30 05:17:54.846646 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:54.846615    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee5dfade-6b22-4653-80d9-75320b4c6117-home\") pod \"test-model-356cae-kserve-78bbbb868f-bmnn8\" (UID: \"ee5dfade-6b22-4653-80d9-75320b4c6117\") " pod="ai-tenant-e2e-inf-a-356cae/test-model-356cae-kserve-78bbbb868f-bmnn8"
Oct 30 05:17:54.846854 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:54.846685    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee5dfade-6b22-4653-80d9-75320b4c6117-model-cache\") pod \"test-model-356cae-kserve-78bbbb868f-bmnn8\" (UID: \"ee5dfade-6b22-4653-80d9-75320b4c6117\") " pod="ai-tenant-e2e-inf-a-356cae/test-model-356cae-kserve-78bbbb868f-bmnn8"
Oct 30 05:17:54.846854 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:54.846726    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee5dfade-6b22-4653-80d9-75320b4c6117-tls-certs\") pod \"test-model-356cae-kserve-78bbbb868f-bmnn8\" (UID: \"ee5dfade-6b22-4653-80d9-75320b4c6117\") " pod="ai-tenant-e2e-inf-a-356cae/test-model-356cae-kserve-78bbbb868f-bmnn8"
Oct 30 05:17:54.846854 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:54.846751    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee5dfade-6b22-4653-80d9-75320b4c6117-tmp-dir\") pod \"test-model-356cae-kserve-78bbbb868f-bmnn8\" (UID: \"ee5dfade-6b22-4653-80d9-75320b4c6117\") " pod="ai-tenant-e2e-inf-a-356cae/test-model-356cae-kserve-78bbbb868f-bmnn8"
Oct 30 05:17:54.847087 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:54.846862    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68zvk\" (UniqueName: \"kubernetes.io/projected/ee5dfade-6b22-4653-80d9-75320b4c6117-kube-api-access-68zvk\") pod \"test-model-356cae-kserve-78bbbb868f-bmnn8\" (UID: \"ee5dfade-6b22-4653-80d9-75320b4c6117\") " pod="ai-tenant-e2e-inf-a-356cae/test-model-356cae-kserve-78bbbb868f-bmnn8"
Oct 30 05:17:54.847087 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:54.846970    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee5dfade-6b22-4653-80d9-75320b4c6117-dshm\") pod \"test-model-356cae-kserve-78bbbb868f-bmnn8\" (UID: \"ee5dfade-6b22-4653-80d9-75320b4c6117\") " pod="ai-tenant-e2e-inf-a-356cae/test-model-356cae-kserve-78bbbb868f-bmnn8"
Oct 30 05:17:54.847200 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:54.847159    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee5dfade-6b22-4653-80d9-75320b4c6117-home\") pod \"test-model-356cae-kserve-78bbbb868f-bmnn8\" (UID: \"ee5dfade-6b22-4653-80d9-75320b4c6117\") " pod="ai-tenant-e2e-inf-a-356cae/test-model-356cae-kserve-78bbbb868f-bmnn8"
Oct 30 05:17:54.847200 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:54.847186    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee5dfade-6b22-4653-80d9-75320b4c6117-model-cache\") pod \"test-model-356cae-kserve-78bbbb868f-bmnn8\" (UID: \"ee5dfade-6b22-4653-80d9-75320b4c6117\") " pod="ai-tenant-e2e-inf-a-356cae/test-model-356cae-kserve-78bbbb868f-bmnn8"
Oct 30 05:17:54.847489 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:54.847461    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee5dfade-6b22-4653-80d9-75320b4c6117-tmp-dir\") pod \"test-model-356cae-kserve-78bbbb868f-bmnn8\" (UID: \"ee5dfade-6b22-4653-80d9-75320b4c6117\") " pod="ai-tenant-e2e-inf-a-356cae/test-model-356cae-kserve-78bbbb868f-bmnn8"
Oct 30 05:17:54.849716 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:54.849685    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee5dfade-6b22-4653-80d9-75320b4c6117-tls-certs\") pod \"test-model-356cae-kserve-78bbbb868f-bmnn8\" (UID: \"ee5dfade-6b22-4653-80d9-75320b4c6117\") " pod="ai-tenant-e2e-inf-a-356cae/test-model-356cae-kserve-78bbbb868f-bmnn8"
Oct 30 05:17:54.862088 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:54.862061    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee5dfade-6b22-4653-80d9-75320b4c6117-dshm\") pod \"test-model-356cae-kserve-78bbbb868f-bmnn8\" (UID: \"ee5dfade-6b22-4653-80d9-75320b4c6117\") " pod="ai-tenant-e2e-inf-a-356cae/test-model-356cae-kserve-78bbbb868f-bmnn8"
Oct 30 05:17:54.891025 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:54.889612    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68zvk\" (UniqueName: \"kubernetes.io/projected/ee5dfade-6b22-4653-80d9-75320b4c6117-kube-api-access-68zvk\") pod \"test-model-356cae-kserve-78bbbb868f-bmnn8\" (UID: \"ee5dfade-6b22-4653-80d9-75320b4c6117\") " pod="ai-tenant-e2e-inf-a-356cae/test-model-356cae-kserve-78bbbb868f-bmnn8"
Oct 30 05:17:54.960928 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:54.950251    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-356cae/test-model-356cae-kserve-78bbbb868f-bmnn8"
Oct 30 05:17:55.334574 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:55.334549    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-356cae/test-model-356cae-kserve-78bbbb868f-bmnn8"]
Oct 30 05:17:55.335766 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:17:55.335738    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee5dfade_6b22_4653_80d9_75320b4c6117.slice/crio-8961abc881ee04d4ab5804ebc1ec16e1abb15f68b41859406859bc9c07fa6ab8 WatchSource:0}: Error finding container 8961abc881ee04d4ab5804ebc1ec16e1abb15f68b41859406859bc9c07fa6ab8: Status 404 returned error can't find the container with id 8961abc881ee04d4ab5804ebc1ec16e1abb15f68b41859406859bc9c07fa6ab8
Oct 30 05:17:55.569245 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:55.569206    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-356cae/test-model-356cae-kserve-78bbbb868f-bmnn8" event={"ID":"ee5dfade-6b22-4653-80d9-75320b4c6117","Type":"ContainerStarted","Data":"f98c5af65814cbcfd8b764f127433a568ac3688d64af0090424eb077e4ca293e"}
Oct 30 05:17:55.569401 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:55.569261    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-356cae/test-model-356cae-kserve-78bbbb868f-bmnn8" event={"ID":"ee5dfade-6b22-4653-80d9-75320b4c6117","Type":"ContainerStarted","Data":"8961abc881ee04d4ab5804ebc1ec16e1abb15f68b41859406859bc9c07fa6ab8"}
Oct 30 05:17:55.609667 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:55.609573    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-356cae/test-model-356cae-kserve-78bbbb868f-bmnn8" podStartSLOduration=1.609559432 podStartE2EDuration="1.609559432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:17:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:17:55.606045578 +0000 UTC m=+2004.829865494" watchObservedRunningTime="2026-10-30 05:17:55.609559432 +0000 UTC m=+2004.833379332"
Oct 30 05:17:55.872237 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:55.872158    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:17:55.971277 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:55.965984    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-356cae/test-model-356cae-kserve-78bbbb868f-bmnn8"
Oct 30 05:17:56.586113 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:56.586081    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-e16297/test-model-e16297-kserve-5654446c8c-lwfxr"]
Oct 30 05:17:56.590843 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:56.590815    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e16297/test-model-e16297-kserve-5654446c8c-lwfxr"
Oct 30 05:17:56.612088 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:56.612058    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-e16297/test-model-e16297-kserve-5654446c8c-lwfxr"]
Oct 30 05:17:56.618133 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:56.618105    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-e16297\"/\"kube-root-ca.crt\""
Oct 30 05:17:56.618275 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:56.618149    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-e16297\"/\"openshift-service-ca.crt\""
Oct 30 05:17:56.618732 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:56.618712    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-e16297\"/\"default-dockercfg-42cqp\""
Oct 30 05:17:56.618834 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:56.618737    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-e16297\"/\"test-model-e16297-kserve-self-signed-certs\""
Oct 30 05:17:56.640866 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:56.640822    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:17:56.783665 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:56.783616    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d4f8e87-1d0e-434a-9be6-d66558d209eb-model-cache\") pod \"test-model-e16297-kserve-5654446c8c-lwfxr\" (UID: \"4d4f8e87-1d0e-434a-9be6-d66558d209eb\") " pod="ai-tenant-e2e-inf-b-e16297/test-model-e16297-kserve-5654446c8c-lwfxr"
Oct 30 05:17:56.783872 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:56.783685    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d4f8e87-1d0e-434a-9be6-d66558d209eb-dshm\") pod \"test-model-e16297-kserve-5654446c8c-lwfxr\" (UID: \"4d4f8e87-1d0e-434a-9be6-d66558d209eb\") " pod="ai-tenant-e2e-inf-b-e16297/test-model-e16297-kserve-5654446c8c-lwfxr"
Oct 30 05:17:56.783872 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:56.783761    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d4f8e87-1d0e-434a-9be6-d66558d209eb-tls-certs\") pod \"test-model-e16297-kserve-5654446c8c-lwfxr\" (UID: \"4d4f8e87-1d0e-434a-9be6-d66558d209eb\") " pod="ai-tenant-e2e-inf-b-e16297/test-model-e16297-kserve-5654446c8c-lwfxr"
Oct 30 05:17:56.784016 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:56.783919    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pwh8\" (UniqueName: \"kubernetes.io/projected/4d4f8e87-1d0e-434a-9be6-d66558d209eb-kube-api-access-5pwh8\") pod \"test-model-e16297-kserve-5654446c8c-lwfxr\" (UID: \"4d4f8e87-1d0e-434a-9be6-d66558d209eb\") " pod="ai-tenant-e2e-inf-b-e16297/test-model-e16297-kserve-5654446c8c-lwfxr"
Oct 30 05:17:56.784016 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:56.783969    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d4f8e87-1d0e-434a-9be6-d66558d209eb-tmp-dir\") pod \"test-model-e16297-kserve-5654446c8c-lwfxr\" (UID: \"4d4f8e87-1d0e-434a-9be6-d66558d209eb\") " pod="ai-tenant-e2e-inf-b-e16297/test-model-e16297-kserve-5654446c8c-lwfxr"
Oct 30 05:17:56.784133 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:56.784118    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d4f8e87-1d0e-434a-9be6-d66558d209eb-home\") pod \"test-model-e16297-kserve-5654446c8c-lwfxr\" (UID: \"4d4f8e87-1d0e-434a-9be6-d66558d209eb\") " pod="ai-tenant-e2e-inf-b-e16297/test-model-e16297-kserve-5654446c8c-lwfxr"
Oct 30 05:17:56.825919 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:17:56.824059    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:17:56.885975 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:56.885815    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d4f8e87-1d0e-434a-9be6-d66558d209eb-model-cache\") pod \"test-model-e16297-kserve-5654446c8c-lwfxr\" (UID: \"4d4f8e87-1d0e-434a-9be6-d66558d209eb\") " pod="ai-tenant-e2e-inf-b-e16297/test-model-e16297-kserve-5654446c8c-lwfxr"
Oct 30 05:17:56.885975 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:56.885885    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d4f8e87-1d0e-434a-9be6-d66558d209eb-dshm\") pod \"test-model-e16297-kserve-5654446c8c-lwfxr\" (UID: \"4d4f8e87-1d0e-434a-9be6-d66558d209eb\") " pod="ai-tenant-e2e-inf-b-e16297/test-model-e16297-kserve-5654446c8c-lwfxr"
Oct 30 05:17:56.886210 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:56.885978    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d4f8e87-1d0e-434a-9be6-d66558d209eb-tls-certs\") pod \"test-model-e16297-kserve-5654446c8c-lwfxr\" (UID: \"4d4f8e87-1d0e-434a-9be6-d66558d209eb\") " pod="ai-tenant-e2e-inf-b-e16297/test-model-e16297-kserve-5654446c8c-lwfxr"
Oct 30 05:17:56.886210 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:56.886091    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pwh8\" (UniqueName: \"kubernetes.io/projected/4d4f8e87-1d0e-434a-9be6-d66558d209eb-kube-api-access-5pwh8\") pod \"test-model-e16297-kserve-5654446c8c-lwfxr\" (UID: \"4d4f8e87-1d0e-434a-9be6-d66558d209eb\") " pod="ai-tenant-e2e-inf-b-e16297/test-model-e16297-kserve-5654446c8c-lwfxr"
Oct 30 05:17:56.886210 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:56.886136    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d4f8e87-1d0e-434a-9be6-d66558d209eb-tmp-dir\") pod \"test-model-e16297-kserve-5654446c8c-lwfxr\" (UID: \"4d4f8e87-1d0e-434a-9be6-d66558d209eb\") " pod="ai-tenant-e2e-inf-b-e16297/test-model-e16297-kserve-5654446c8c-lwfxr"
Oct 30 05:17:56.886367 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:56.886248    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d4f8e87-1d0e-434a-9be6-d66558d209eb-home\") pod \"test-model-e16297-kserve-5654446c8c-lwfxr\" (UID: \"4d4f8e87-1d0e-434a-9be6-d66558d209eb\") " pod="ai-tenant-e2e-inf-b-e16297/test-model-e16297-kserve-5654446c8c-lwfxr"
Oct 30 05:17:56.890926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:56.886740    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d4f8e87-1d0e-434a-9be6-d66558d209eb-home\") pod \"test-model-e16297-kserve-5654446c8c-lwfxr\" (UID: \"4d4f8e87-1d0e-434a-9be6-d66558d209eb\") " pod="ai-tenant-e2e-inf-b-e16297/test-model-e16297-kserve-5654446c8c-lwfxr"
Oct 30 05:17:56.890926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:56.886993    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d4f8e87-1d0e-434a-9be6-d66558d209eb-model-cache\") pod \"test-model-e16297-kserve-5654446c8c-lwfxr\" (UID: \"4d4f8e87-1d0e-434a-9be6-d66558d209eb\") " pod="ai-tenant-e2e-inf-b-e16297/test-model-e16297-kserve-5654446c8c-lwfxr"
Oct 30 05:17:56.890926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:56.888119    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d4f8e87-1d0e-434a-9be6-d66558d209eb-tmp-dir\") pod \"test-model-e16297-kserve-5654446c8c-lwfxr\" (UID: \"4d4f8e87-1d0e-434a-9be6-d66558d209eb\") " pod="ai-tenant-e2e-inf-b-e16297/test-model-e16297-kserve-5654446c8c-lwfxr"
Oct 30 05:17:56.915976 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:56.915512    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d4f8e87-1d0e-434a-9be6-d66558d209eb-dshm\") pod \"test-model-e16297-kserve-5654446c8c-lwfxr\" (UID: \"4d4f8e87-1d0e-434a-9be6-d66558d209eb\") " pod="ai-tenant-e2e-inf-b-e16297/test-model-e16297-kserve-5654446c8c-lwfxr"
Oct 30 05:17:56.926184 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:56.926083    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pwh8\" (UniqueName: \"kubernetes.io/projected/4d4f8e87-1d0e-434a-9be6-d66558d209eb-kube-api-access-5pwh8\") pod \"test-model-e16297-kserve-5654446c8c-lwfxr\" (UID: \"4d4f8e87-1d0e-434a-9be6-d66558d209eb\") " pod="ai-tenant-e2e-inf-b-e16297/test-model-e16297-kserve-5654446c8c-lwfxr"
Oct 30 05:17:56.927149 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:56.927105    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d4f8e87-1d0e-434a-9be6-d66558d209eb-tls-certs\") pod \"test-model-e16297-kserve-5654446c8c-lwfxr\" (UID: \"4d4f8e87-1d0e-434a-9be6-d66558d209eb\") " pod="ai-tenant-e2e-inf-b-e16297/test-model-e16297-kserve-5654446c8c-lwfxr"
Oct 30 05:17:57.205787 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:57.205706    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e16297/test-model-e16297-kserve-5654446c8c-lwfxr"
Oct 30 05:17:57.515651 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:57.515621    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-e16297/test-model-e16297-kserve-5654446c8c-lwfxr"]
Oct 30 05:17:57.517347 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:17:57.517320    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d4f8e87_1d0e_434a_9be6_d66558d209eb.slice/crio-add869577c7a4990b5cfabbcf0677472a78203d53f5c8b97bfa6cf1635bbcf3c WatchSource:0}: Error finding container add869577c7a4990b5cfabbcf0677472a78203d53f5c8b97bfa6cf1635bbcf3c: Status 404 returned error can't find the container with id add869577c7a4990b5cfabbcf0677472a78203d53f5c8b97bfa6cf1635bbcf3c
Oct 30 05:17:57.588566 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:57.588532    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e16297/test-model-e16297-kserve-5654446c8c-lwfxr" event={"ID":"4d4f8e87-1d0e-434a-9be6-d66558d209eb","Type":"ContainerStarted","Data":"da4011e3554a4183ea205f92f3fd34ff8e6e258c47feee5227a06f95e844ea1d"}
Oct 30 05:17:57.588936 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:57.588585    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e16297/test-model-e16297-kserve-5654446c8c-lwfxr" event={"ID":"4d4f8e87-1d0e-434a-9be6-d66558d209eb","Type":"ContainerStarted","Data":"add869577c7a4990b5cfabbcf0677472a78203d53f5c8b97bfa6cf1635bbcf3c"}
Oct 30 05:17:57.631242 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:57.631188    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-e16297/test-model-e16297-kserve-5654446c8c-lwfxr" podStartSLOduration=1.6311745 podStartE2EDuration="1.6311745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:17:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:17:57.625963114 +0000 UTC m=+2006.849783015" watchObservedRunningTime="2026-10-30 05:17:57.6311745 +0000 UTC m=+2006.854994400"
Oct 30 05:17:57.755322 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:57.755290    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:17:58.206053 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:58.206022    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-e16297/test-model-e16297-kserve-5654446c8c-lwfxr"
Oct 30 05:17:59.495381 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:59.495348    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:17:59.514589 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:59.514556    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw"]
Oct 30 05:17:59.518981 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:59.518961    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw"
Oct 30 05:17:59.523677 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:59.523651    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aed7491-445f-4c4a-86c1-3c844692deee-workload-socket\") pod \"e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw\" (UID: \"8aed7491-445f-4c4a-86c1-3c844692deee\") " pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw"
Oct 30 05:17:59.523821 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:59.523706    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aed7491-445f-4c4a-86c1-3c844692deee-istiod-ca-cert\") pod \"e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw\" (UID: \"8aed7491-445f-4c4a-86c1-3c844692deee\") " pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw"
Oct 30 05:17:59.523821 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:59.523770    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aed7491-445f-4c4a-86c1-3c844692deee-istio-podinfo\") pod \"e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw\" (UID: \"8aed7491-445f-4c4a-86c1-3c844692deee\") " pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw"
Oct 30 05:17:59.523821 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:59.523813    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aed7491-445f-4c4a-86c1-3c844692deee-workload-certs\") pod \"e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw\" (UID: \"8aed7491-445f-4c4a-86c1-3c844692deee\") " pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw"
Oct 30 05:17:59.524015 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:59.523853    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8aed7491-445f-4c4a-86c1-3c844692deee-istio-token\") pod \"e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw\" (UID: \"8aed7491-445f-4c4a-86c1-3c844692deee\") " pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw"
Oct 30 05:17:59.524015 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:59.523881    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aed7491-445f-4c4a-86c1-3c844692deee-istio-data\") pod \"e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw\" (UID: \"8aed7491-445f-4c4a-86c1-3c844692deee\") " pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw"
Oct 30 05:17:59.524109 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:59.524023    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aed7491-445f-4c4a-86c1-3c844692deee-istio-envoy\") pod \"e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw\" (UID: \"8aed7491-445f-4c4a-86c1-3c844692deee\") " pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw"
Oct 30 05:17:59.524166 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:59.524144    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4mxl\" (UniqueName: \"kubernetes.io/projected/8aed7491-445f-4c4a-86c1-3c844692deee-kube-api-access-m4mxl\") pod \"e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw\" (UID: \"8aed7491-445f-4c4a-86c1-3c844692deee\") " pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw"
Oct 30 05:17:59.524217 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:59.524186    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aed7491-445f-4c4a-86c1-3c844692deee-credential-socket\") pod \"e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw\" (UID: \"8aed7491-445f-4c4a-86c1-3c844692deee\") " pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw"
Oct 30 05:17:59.537594 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:59.537572    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw"]
Oct 30 05:17:59.557742 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:59.543132    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-36fd3c-openshift-default-dockercfg-5l4n9\""
Oct 30 05:17:59.625086 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:59.625053    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aed7491-445f-4c4a-86c1-3c844692deee-workload-certs\") pod \"e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw\" (UID: \"8aed7491-445f-4c4a-86c1-3c844692deee\") " pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw"
Oct 30 05:17:59.625264 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:59.625214    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8aed7491-445f-4c4a-86c1-3c844692deee-istio-token\") pod \"e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw\" (UID: \"8aed7491-445f-4c4a-86c1-3c844692deee\") " pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw"
Oct 30 05:17:59.625354 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:59.625270    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aed7491-445f-4c4a-86c1-3c844692deee-istio-data\") pod \"e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw\" (UID: \"8aed7491-445f-4c4a-86c1-3c844692deee\") " pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw"
Oct 30 05:17:59.625354 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:59.625333    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aed7491-445f-4c4a-86c1-3c844692deee-istio-envoy\") pod \"e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw\" (UID: \"8aed7491-445f-4c4a-86c1-3c844692deee\") " pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw"
Oct 30 05:17:59.625457 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:59.625430    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4mxl\" (UniqueName: \"kubernetes.io/projected/8aed7491-445f-4c4a-86c1-3c844692deee-kube-api-access-m4mxl\") pod \"e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw\" (UID: \"8aed7491-445f-4c4a-86c1-3c844692deee\") " pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw"
Oct 30 05:17:59.625496 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:59.625467    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aed7491-445f-4c4a-86c1-3c844692deee-credential-socket\") pod \"e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw\" (UID: \"8aed7491-445f-4c4a-86c1-3c844692deee\") " pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw"
Oct 30 05:17:59.625546 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:59.625432    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aed7491-445f-4c4a-86c1-3c844692deee-workload-certs\") pod \"e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw\" (UID: \"8aed7491-445f-4c4a-86c1-3c844692deee\") " pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw"
Oct 30 05:17:59.625641 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:59.625618    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aed7491-445f-4c4a-86c1-3c844692deee-workload-socket\") pod \"e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw\" (UID: \"8aed7491-445f-4c4a-86c1-3c844692deee\") " pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw"
Oct 30 05:17:59.625710 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:59.625662    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aed7491-445f-4c4a-86c1-3c844692deee-istio-data\") pod \"e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw\" (UID: \"8aed7491-445f-4c4a-86c1-3c844692deee\") " pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw"
Oct 30 05:17:59.625754 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:59.625725    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aed7491-445f-4c4a-86c1-3c844692deee-istiod-ca-cert\") pod \"e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw\" (UID: \"8aed7491-445f-4c4a-86c1-3c844692deee\") " pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw"
Oct 30 05:17:59.625817 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:59.625776    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aed7491-445f-4c4a-86c1-3c844692deee-istio-podinfo\") pod \"e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw\" (UID: \"8aed7491-445f-4c4a-86c1-3c844692deee\") " pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw"
Oct 30 05:17:59.626062 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:59.625924    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aed7491-445f-4c4a-86c1-3c844692deee-workload-socket\") pod \"e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw\" (UID: \"8aed7491-445f-4c4a-86c1-3c844692deee\") " pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw"
Oct 30 05:17:59.626062 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:59.625944    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aed7491-445f-4c4a-86c1-3c844692deee-credential-socket\") pod \"e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw\" (UID: \"8aed7491-445f-4c4a-86c1-3c844692deee\") " pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw"
Oct 30 05:17:59.626546 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:59.626520    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aed7491-445f-4c4a-86c1-3c844692deee-istiod-ca-cert\") pod \"e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw\" (UID: \"8aed7491-445f-4c4a-86c1-3c844692deee\") " pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw"
Oct 30 05:17:59.630336 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:59.630312    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aed7491-445f-4c4a-86c1-3c844692deee-istio-podinfo\") pod \"e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw\" (UID: \"8aed7491-445f-4c4a-86c1-3c844692deee\") " pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw"
Oct 30 05:17:59.640470 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:59.640442    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aed7491-445f-4c4a-86c1-3c844692deee-istio-envoy\") pod \"e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw\" (UID: \"8aed7491-445f-4c4a-86c1-3c844692deee\") " pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw"
Oct 30 05:17:59.644790 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:59.644763    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8aed7491-445f-4c4a-86c1-3c844692deee-istio-token\") pod \"e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw\" (UID: \"8aed7491-445f-4c4a-86c1-3c844692deee\") " pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw"
Oct 30 05:17:59.646158 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:59.646139    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4mxl\" (UniqueName: \"kubernetes.io/projected/8aed7491-445f-4c4a-86c1-3c844692deee-kube-api-access-m4mxl\") pod \"e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw\" (UID: \"8aed7491-445f-4c4a-86c1-3c844692deee\") " pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw"
Oct 30 05:17:59.836431 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:17:59.836401    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw"
Oct 30 05:18:00.153413 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:00.153380    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw"]
Oct 30 05:18:00.153611 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:18:00.153588    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aed7491_445f_4c4a_86c1_3c844692deee.slice/crio-ab0233fb00b1fb6a00d237b606db244cbcbcf363530d47ed2e9fd8cb88eb4c7e WatchSource:0}: Error finding container ab0233fb00b1fb6a00d237b606db244cbcbcf363530d47ed2e9fd8cb88eb4c7e: Status 404 returned error can't find the container with id ab0233fb00b1fb6a00d237b606db244cbcbcf363530d47ed2e9fd8cb88eb4c7e
Oct 30 05:18:00.156699 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:00.156664    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:18:00.156800 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:00.156744    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:18:00.156800 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:00.156790    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:18:00.621139 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:00.621100    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw" event={"ID":"8aed7491-445f-4c4a-86c1-3c844692deee","Type":"ContainerStarted","Data":"54f3d0f249e07023ad63a3e0495ab1f3504347e51b5c2a8e4f0067dad6268861"}
Oct 30 05:18:00.621636 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:00.621167    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw" event={"ID":"8aed7491-445f-4c4a-86c1-3c844692deee","Type":"ContainerStarted","Data":"ab0233fb00b1fb6a00d237b606db244cbcbcf363530d47ed2e9fd8cb88eb4c7e"}
Oct 30 05:18:00.672536 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:00.672488    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw" podStartSLOduration=1.672474354 podStartE2EDuration="1.672474354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:17:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:18:00.671500088 +0000 UTC m=+2009.895320015" watchObservedRunningTime="2026-10-30 05:18:00.672474354 +0000 UTC m=+2009.896294254"
Oct 30 05:18:00.836643 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:00.836611    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw"
Oct 30 05:18:01.108842 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:01.108809    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:18:01.838293 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:01.838248    5898 patch_prober.go:29] interesting pod/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 05:18:01.838715 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:01.838321    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw" podUID="8aed7491-445f-4c4a-86c1-3c844692deee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 05:18:02.837713 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:02.837681    5898 patch_prober.go:29] interesting pod/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 05:18:02.837946 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:02.837749    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw" podUID="8aed7491-445f-4c4a-86c1-3c844692deee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 05:18:02.952813 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:02.952775    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:18:03.837056 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:03.837023    5898 patch_prober.go:29] interesting pod/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 05:18:03.837243 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:03.837082    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw" podUID="8aed7491-445f-4c4a-86c1-3c844692deee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 05:18:04.686932 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:04.686339    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:18:04.838016 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:04.837986    5898 patch_prober.go:29] interesting pod/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 05:18:04.838185 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:04.838034    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw" podUID="8aed7491-445f-4c4a-86c1-3c844692deee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 05:18:04.954019 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:04.953931    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-356cae/test-model-356cae-kserve-78bbbb868f-bmnn8"
Oct 30 05:18:04.966743 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:04.966719    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-356cae/test-model-356cae-kserve-78bbbb868f-bmnn8"
Oct 30 05:18:05.706206 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:05.706169    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-356cae/test-model-356cae-kserve-78bbbb868f-bmnn8"
Oct 30 05:18:05.838395 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:05.838363    5898 patch_prober.go:29] interesting pod/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 05:18:05.838568 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:05.838419    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw" podUID="8aed7491-445f-4c4a-86c1-3c844692deee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 05:18:06.317797 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:06.317760    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:18:06.827969 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:06.827935    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:18:06.842057 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:06.842031    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw"
Oct 30 05:18:06.842496 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:06.842483    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw"
Oct 30 05:18:06.843129 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:06.843111    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw"
Oct 30 05:18:07.206588 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:07.206515    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-e16297/test-model-e16297-kserve-5654446c8c-lwfxr"
Oct 30 05:18:07.218756 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:07.218731    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-e16297/test-model-e16297-kserve-5654446c8c-lwfxr"
Oct 30 05:18:07.698106 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:07.698077    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-e16297/test-model-e16297-kserve-5654446c8c-lwfxr"
Oct 30 05:18:08.054587 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:08.054505    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:18:09.727934 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:09.727862    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:18:11.581630 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:11.581594    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:18:11.994403 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:11.994315    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:18:12.223924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:12.222053    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:18:12.675896 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:12.675856    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:18:13.165763 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:13.165728    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:18:13.337376 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:13.337335    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:18:13.646839 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:13.646801    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:18:15.252723 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:15.252669    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:18:17.066609 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:17.066570    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:18:17.353118 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:17.353083    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:18:17.453612 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:17.453563    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:18:19.060398 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:19.060367    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:18:19.160066 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:19.160036    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:18:19.271507 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:19.271473    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:18:20.919804 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:20.919771    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:18:22.525414 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:22.525379    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:18:24.745925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:24.745470    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:18:25.326364 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:25.326167    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-36fd3c/test-model-36fd3c-kserve-86d45f7b9c-qhhbc"]
Oct 30 05:18:25.353946 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:25.353507    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:18:26.386386 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:26.386348    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:18:26.499285 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:26.499251    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:18:26.871726 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:18:26.871687    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:18:26.980928 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:26.979370    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:18:27.813030 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:27.812987    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:18:29.076927 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:29.075474    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:18:30.732811 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:30.732700    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:18:32.256028 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:32.255991    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js"]
Oct 30 05:18:32.261974 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:32.261951    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js"
Oct 30 05:18:32.279086 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:32.279050    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js"]
Oct 30 05:18:32.372215 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:32.372186    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-a914e9-openshift-default-dockercfg-zl54b\""
Oct 30 05:18:32.381027 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:32.380994    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbe41d7a-fb7d-4712-ae76-19d311778e9f-workload-socket\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js\" (UID: \"fbe41d7a-fb7d-4712-ae76-19d311778e9f\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js"
Oct 30 05:18:32.381191 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:32.381037    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbe41d7a-fb7d-4712-ae76-19d311778e9f-credential-socket\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js\" (UID: \"fbe41d7a-fb7d-4712-ae76-19d311778e9f\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js"
Oct 30 05:18:32.381191 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:32.381162    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzbg2\" (UniqueName: \"kubernetes.io/projected/fbe41d7a-fb7d-4712-ae76-19d311778e9f-kube-api-access-zzbg2\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js\" (UID: \"fbe41d7a-fb7d-4712-ae76-19d311778e9f\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js"
Oct 30 05:18:32.381278 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:32.381190    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbe41d7a-fb7d-4712-ae76-19d311778e9f-istiod-ca-cert\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js\" (UID: \"fbe41d7a-fb7d-4712-ae76-19d311778e9f\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js"
Oct 30 05:18:32.381333 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:32.381312    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbe41d7a-fb7d-4712-ae76-19d311778e9f-istio-token\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js\" (UID: \"fbe41d7a-fb7d-4712-ae76-19d311778e9f\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js"
Oct 30 05:18:32.381385 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:32.381362    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbe41d7a-fb7d-4712-ae76-19d311778e9f-istio-envoy\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js\" (UID: \"fbe41d7a-fb7d-4712-ae76-19d311778e9f\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js"
Oct 30 05:18:32.381492 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:32.381468    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbe41d7a-fb7d-4712-ae76-19d311778e9f-workload-certs\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js\" (UID: \"fbe41d7a-fb7d-4712-ae76-19d311778e9f\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js"
Oct 30 05:18:32.381646 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:32.381508    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbe41d7a-fb7d-4712-ae76-19d311778e9f-istio-podinfo\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js\" (UID: \"fbe41d7a-fb7d-4712-ae76-19d311778e9f\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js"
Oct 30 05:18:32.381646 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:32.381640    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbe41d7a-fb7d-4712-ae76-19d311778e9f-istio-data\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js\" (UID: \"fbe41d7a-fb7d-4712-ae76-19d311778e9f\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js"
Oct 30 05:18:32.483196 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:32.483164    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbe41d7a-fb7d-4712-ae76-19d311778e9f-istio-token\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js\" (UID: \"fbe41d7a-fb7d-4712-ae76-19d311778e9f\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js"
Oct 30 05:18:32.483393 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:32.483249    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbe41d7a-fb7d-4712-ae76-19d311778e9f-istio-envoy\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js\" (UID: \"fbe41d7a-fb7d-4712-ae76-19d311778e9f\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js"
Oct 30 05:18:32.483393 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:32.483299    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbe41d7a-fb7d-4712-ae76-19d311778e9f-workload-certs\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js\" (UID: \"fbe41d7a-fb7d-4712-ae76-19d311778e9f\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js"
Oct 30 05:18:32.483393 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:32.483330    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbe41d7a-fb7d-4712-ae76-19d311778e9f-istio-podinfo\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js\" (UID: \"fbe41d7a-fb7d-4712-ae76-19d311778e9f\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js"
Oct 30 05:18:32.483618 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:32.483429    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbe41d7a-fb7d-4712-ae76-19d311778e9f-istio-data\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js\" (UID: \"fbe41d7a-fb7d-4712-ae76-19d311778e9f\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js"
Oct 30 05:18:32.483618 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:32.483541    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbe41d7a-fb7d-4712-ae76-19d311778e9f-workload-socket\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js\" (UID: \"fbe41d7a-fb7d-4712-ae76-19d311778e9f\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js"
Oct 30 05:18:32.483618 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:32.483581    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbe41d7a-fb7d-4712-ae76-19d311778e9f-credential-socket\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js\" (UID: \"fbe41d7a-fb7d-4712-ae76-19d311778e9f\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js"
Oct 30 05:18:32.483786 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:32.483636    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzbg2\" (UniqueName: \"kubernetes.io/projected/fbe41d7a-fb7d-4712-ae76-19d311778e9f-kube-api-access-zzbg2\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js\" (UID: \"fbe41d7a-fb7d-4712-ae76-19d311778e9f\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js"
Oct 30 05:18:32.483786 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:32.483674    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbe41d7a-fb7d-4712-ae76-19d311778e9f-istiod-ca-cert\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js\" (UID: \"fbe41d7a-fb7d-4712-ae76-19d311778e9f\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js"
Oct 30 05:18:32.483894 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:32.483816    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbe41d7a-fb7d-4712-ae76-19d311778e9f-workload-certs\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js\" (UID: \"fbe41d7a-fb7d-4712-ae76-19d311778e9f\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js"
Oct 30 05:18:32.483894 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:32.483832    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbe41d7a-fb7d-4712-ae76-19d311778e9f-istio-data\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js\" (UID: \"fbe41d7a-fb7d-4712-ae76-19d311778e9f\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js"
Oct 30 05:18:32.484024 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:32.483940    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbe41d7a-fb7d-4712-ae76-19d311778e9f-workload-socket\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js\" (UID: \"fbe41d7a-fb7d-4712-ae76-19d311778e9f\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js"
Oct 30 05:18:32.484141 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:32.484121    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbe41d7a-fb7d-4712-ae76-19d311778e9f-credential-socket\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js\" (UID: \"fbe41d7a-fb7d-4712-ae76-19d311778e9f\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js"
Oct 30 05:18:32.484563 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:32.484536    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbe41d7a-fb7d-4712-ae76-19d311778e9f-istiod-ca-cert\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js\" (UID: \"fbe41d7a-fb7d-4712-ae76-19d311778e9f\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js"
Oct 30 05:18:32.485787 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:32.485761    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbe41d7a-fb7d-4712-ae76-19d311778e9f-istio-envoy\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js\" (UID: \"fbe41d7a-fb7d-4712-ae76-19d311778e9f\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js"
Oct 30 05:18:32.486173 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:32.486154    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbe41d7a-fb7d-4712-ae76-19d311778e9f-istio-podinfo\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js\" (UID: \"fbe41d7a-fb7d-4712-ae76-19d311778e9f\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js"
Oct 30 05:18:32.504652 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:32.504622    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbe41d7a-fb7d-4712-ae76-19d311778e9f-istio-token\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js\" (UID: \"fbe41d7a-fb7d-4712-ae76-19d311778e9f\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js"
Oct 30 05:18:32.504989 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:32.504968    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzbg2\" (UniqueName: \"kubernetes.io/projected/fbe41d7a-fb7d-4712-ae76-19d311778e9f-kube-api-access-zzbg2\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js\" (UID: \"fbe41d7a-fb7d-4712-ae76-19d311778e9f\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js"
Oct 30 05:18:32.548297 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:32.548217    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:18:32.594866 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:32.594834    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js"
Oct 30 05:18:32.964233 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:32.964200    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js"]
Oct 30 05:18:32.966587 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:18:32.966561    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbe41d7a_fb7d_4712_ae76_19d311778e9f.slice/crio-b473411c71bdfbe0c2fd237ac6a59f05a14892e4a480f40734a4b8b2c370462a WatchSource:0}: Error finding container b473411c71bdfbe0c2fd237ac6a59f05a14892e4a480f40734a4b8b2c370462a: Status 404 returned error can't find the container with id b473411c71bdfbe0c2fd237ac6a59f05a14892e4a480f40734a4b8b2c370462a
Oct 30 05:18:32.970103 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:32.970069    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:18:32.970200 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:32.970152    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:18:32.970200 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:32.970182    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:18:33.119700 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:33.119655    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js" event={"ID":"fbe41d7a-fb7d-4712-ae76-19d311778e9f","Type":"ContainerStarted","Data":"7c844efcd3083d3eb575bc2f327eb86dabe53d1af849ad9a084e433834c43176"}
Oct 30 05:18:33.119926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:33.119726    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js" event={"ID":"fbe41d7a-fb7d-4712-ae76-19d311778e9f","Type":"ContainerStarted","Data":"b473411c71bdfbe0c2fd237ac6a59f05a14892e4a480f40734a4b8b2c370462a"}
Oct 30 05:18:33.175847 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:33.175793    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js" podStartSLOduration=1.175777806 podStartE2EDuration="1.175777806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:18:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:18:33.170769845 +0000 UTC m=+2042.394589788" watchObservedRunningTime="2026-10-30 05:18:33.175777806 +0000 UTC m=+2042.399597706"
Oct 30 05:18:33.595714 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:33.595675    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js"
Oct 30 05:18:34.404941 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:34.404352    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:18:34.599924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:34.599579    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Oct 30 05:18:34.599924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:34.599640    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js" podUID="fbe41d7a-fb7d-4712-ae76-19d311778e9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Oct 30 05:18:35.138048 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:35.138013    5898 generic.go:356] "Generic (PLEG): container finished" podID="8a999b0b-fe75-49de-af7f-bdc3145966c7" containerID="59e7172c0e4cddcdc33c748a06b25fba80835c7d5e4cd741ef64bc5dfe9b03e5" exitCode=0
Oct 30 05:18:35.138297 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:35.138088    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w" event={"ID":"8a999b0b-fe75-49de-af7f-bdc3145966c7","Type":"ContainerDied","Data":"59e7172c0e4cddcdc33c748a06b25fba80835c7d5e4cd741ef64bc5dfe9b03e5"}
Oct 30 05:18:35.138629 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:35.138612    5898 scope.go:122] "RemoveContainer" containerID="59e7172c0e4cddcdc33c748a06b25fba80835c7d5e4cd741ef64bc5dfe9b03e5"
Oct 30 05:18:35.140302 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:35.140277    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:18:35.140393 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:35.140343    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:18:35.140393 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:35.140374    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:18:35.596923 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:35.596240    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Oct 30 05:18:35.596923 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:35.596310    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js" podUID="fbe41d7a-fb7d-4712-ae76-19d311778e9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Oct 30 05:18:36.152572 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:36.152537    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:18:36.159200 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:36.159158    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w" event={"ID":"8a999b0b-fe75-49de-af7f-bdc3145966c7","Type":"ContainerStarted","Data":"8a4efacddfe6b2ef605af488de8353636890bc5064c7a6ecb722164682c61bb5"}
Oct 30 05:18:36.250024 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:36.249834    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:18:36.337927 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:36.336252    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w"
Oct 30 05:18:36.372526 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:36.372485    5898 patch_prober.go:29] interesting pod/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 30 05:18:36.372748 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:36.372554    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w" podUID="8a999b0b-fe75-49de-af7f-bdc3145966c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 30 05:18:36.609370 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:36.609335    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Oct 30 05:18:36.609659 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:36.609633    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js" podUID="fbe41d7a-fb7d-4712-ae76-19d311778e9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Oct 30 05:18:37.332920 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:37.332343    5898 patch_prober.go:29] interesting pod/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 30 05:18:37.332920 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:37.332398    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w" podUID="8a999b0b-fe75-49de-af7f-bdc3145966c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 30 05:18:37.598983 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:37.597997    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Oct 30 05:18:37.598983 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:37.598061    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js" podUID="fbe41d7a-fb7d-4712-ae76-19d311778e9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Oct 30 05:18:38.136945 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:38.136634    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w"
Oct 30 05:18:38.225447 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:38.225399    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:18:38.348928 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:38.343826    5898 patch_prober.go:29] interesting pod/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 30 05:18:38.348928 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:38.343889    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w" podUID="8a999b0b-fe75-49de-af7f-bdc3145966c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 30 05:18:38.633356 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:38.632794    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Oct 30 05:18:38.633356 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:38.632869    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js" podUID="fbe41d7a-fb7d-4712-ae76-19d311778e9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Oct 30 05:18:39.332451 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:39.332408    5898 patch_prober.go:29] interesting pod/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 30 05:18:39.332631 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:39.332472    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w" podUID="8a999b0b-fe75-49de-af7f-bdc3145966c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 30 05:18:39.716390 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:39.715926    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js"
Oct 30 05:18:39.717477 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:39.717457    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js"
Oct 30 05:18:39.718262 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:39.718240    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js"
Oct 30 05:18:39.966780 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:39.966701    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:18:40.332887 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:40.332857    5898 patch_prober.go:29] interesting pod/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 30 05:18:40.333104 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:40.332936    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w" podUID="8a999b0b-fe75-49de-af7f-bdc3145966c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 30 05:18:40.509258 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:40.509213    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-36fd3c/test-model-36fd3c-kserve-86d45f7b9c-qhhbc" podUID="da336e19-5be4-46ea-80c9-1541cacfa7f0" containerName="main" containerID="cri-o://9b049094af0e563a7941c8d609d7635c3ffdcf2cca5c3ce34c3a1fea7572d78f" gracePeriod=45
Oct 30 05:18:40.846253 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:40.846230    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-36fd3c/test-model-36fd3c-kserve-86d45f7b9c-qhhbc"
Oct 30 05:18:40.914355 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:40.913016    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da336e19-5be4-46ea-80c9-1541cacfa7f0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da336e19-5be4-46ea-80c9-1541cacfa7f0-model-cache\") pod \"da336e19-5be4-46ea-80c9-1541cacfa7f0\" (UID: \"da336e19-5be4-46ea-80c9-1541cacfa7f0\") "
Oct 30 05:18:40.914355 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:40.913084    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da336e19-5be4-46ea-80c9-1541cacfa7f0-tls-certs\" (UniqueName: \"kubernetes.io/secret/da336e19-5be4-46ea-80c9-1541cacfa7f0-tls-certs\") pod \"da336e19-5be4-46ea-80c9-1541cacfa7f0\" (UID: \"da336e19-5be4-46ea-80c9-1541cacfa7f0\") "
Oct 30 05:18:40.914355 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:40.913311    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da336e19-5be4-46ea-80c9-1541cacfa7f0-kube-api-access-cmb5j\" (UniqueName: \"kubernetes.io/projected/da336e19-5be4-46ea-80c9-1541cacfa7f0-kube-api-access-cmb5j\") pod \"da336e19-5be4-46ea-80c9-1541cacfa7f0\" (UID: \"da336e19-5be4-46ea-80c9-1541cacfa7f0\") "
Oct 30 05:18:40.914355 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:40.913406    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da336e19-5be4-46ea-80c9-1541cacfa7f0-home\" (UniqueName: \"kubernetes.io/empty-dir/da336e19-5be4-46ea-80c9-1541cacfa7f0-home\") pod \"da336e19-5be4-46ea-80c9-1541cacfa7f0\" (UID: \"da336e19-5be4-46ea-80c9-1541cacfa7f0\") "
Oct 30 05:18:40.914355 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:40.913570    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da336e19-5be4-46ea-80c9-1541cacfa7f0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/da336e19-5be4-46ea-80c9-1541cacfa7f0-dshm\") pod \"da336e19-5be4-46ea-80c9-1541cacfa7f0\" (UID: \"da336e19-5be4-46ea-80c9-1541cacfa7f0\") "
Oct 30 05:18:40.914355 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:40.913705    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da336e19-5be4-46ea-80c9-1541cacfa7f0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da336e19-5be4-46ea-80c9-1541cacfa7f0-tmp-dir\") pod \"da336e19-5be4-46ea-80c9-1541cacfa7f0\" (UID: \"da336e19-5be4-46ea-80c9-1541cacfa7f0\") "
Oct 30 05:18:40.918164 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:40.916014    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da336e19-5be4-46ea-80c9-1541cacfa7f0-model-cache" pod "da336e19-5be4-46ea-80c9-1541cacfa7f0" (UID: "da336e19-5be4-46ea-80c9-1541cacfa7f0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:18:40.929021 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:40.927829    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da336e19-5be4-46ea-80c9-1541cacfa7f0-tmp-dir" pod "da336e19-5be4-46ea-80c9-1541cacfa7f0" (UID: "da336e19-5be4-46ea-80c9-1541cacfa7f0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:18:40.929678 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:40.929654    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da336e19-5be4-46ea-80c9-1541cacfa7f0-home" pod "da336e19-5be4-46ea-80c9-1541cacfa7f0" (UID: "da336e19-5be4-46ea-80c9-1541cacfa7f0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:18:40.932462 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:40.932439    5898 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da336e19-5be4-46ea-80c9-1541cacfa7f0-model-cache\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:18:40.933086 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:40.933055    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da336e19-5be4-46ea-80c9-1541cacfa7f0-dshm" pod "da336e19-5be4-46ea-80c9-1541cacfa7f0" (UID: "da336e19-5be4-46ea-80c9-1541cacfa7f0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:18:40.936834 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:40.936802    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da336e19-5be4-46ea-80c9-1541cacfa7f0-tls-certs" pod "da336e19-5be4-46ea-80c9-1541cacfa7f0" (UID: "da336e19-5be4-46ea-80c9-1541cacfa7f0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:18:40.942060 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:40.941984    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da336e19-5be4-46ea-80c9-1541cacfa7f0-kube-api-access-cmb5j" pod "da336e19-5be4-46ea-80c9-1541cacfa7f0" (UID: "da336e19-5be4-46ea-80c9-1541cacfa7f0"). InnerVolumeSpecName "kube-api-access-cmb5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:18:41.033589 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:41.033555    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmb5j\" (UniqueName: \"kubernetes.io/projected/da336e19-5be4-46ea-80c9-1541cacfa7f0-kube-api-access-cmb5j\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:18:41.033589 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:41.033584    5898 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da336e19-5be4-46ea-80c9-1541cacfa7f0-home\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:18:41.033589 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:41.033594    5898 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da336e19-5be4-46ea-80c9-1541cacfa7f0-dshm\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:18:41.033811 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:41.033603    5898 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da336e19-5be4-46ea-80c9-1541cacfa7f0-tmp-dir\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:18:41.033811 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:41.033611    5898 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da336e19-5be4-46ea-80c9-1541cacfa7f0-tls-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:18:41.225570 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:41.225489    5898 generic.go:356] "Generic (PLEG): container finished" podID="da336e19-5be4-46ea-80c9-1541cacfa7f0" containerID="9b049094af0e563a7941c8d609d7635c3ffdcf2cca5c3ce34c3a1fea7572d78f" exitCode=0
Oct 30 05:18:41.225570 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:41.225559    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-36fd3c/test-model-36fd3c-kserve-86d45f7b9c-qhhbc"
Oct 30 05:18:41.225760 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:41.225569    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-36fd3c/test-model-36fd3c-kserve-86d45f7b9c-qhhbc" event={"ID":"da336e19-5be4-46ea-80c9-1541cacfa7f0","Type":"ContainerDied","Data":"9b049094af0e563a7941c8d609d7635c3ffdcf2cca5c3ce34c3a1fea7572d78f"}
Oct 30 05:18:41.225760 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:41.225628    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-36fd3c/test-model-36fd3c-kserve-86d45f7b9c-qhhbc" event={"ID":"da336e19-5be4-46ea-80c9-1541cacfa7f0","Type":"ContainerDied","Data":"f3eb37376825b6427b4fe546ee1ba0df45dcf69bfbfb31ca82f835a39c75802e"}
Oct 30 05:18:41.225760 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:41.225659    5898 scope.go:122] "RemoveContainer" containerID="9b049094af0e563a7941c8d609d7635c3ffdcf2cca5c3ce34c3a1fea7572d78f"
Oct 30 05:18:41.238949 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:41.238923    5898 scope.go:122] "RemoveContainer" containerID="9b049094af0e563a7941c8d609d7635c3ffdcf2cca5c3ce34c3a1fea7572d78f"
Oct 30 05:18:41.239273 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:18:41.239244    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b049094af0e563a7941c8d609d7635c3ffdcf2cca5c3ce34c3a1fea7572d78f\": container with ID starting with 9b049094af0e563a7941c8d609d7635c3ffdcf2cca5c3ce34c3a1fea7572d78f not found: ID does not exist" containerID="9b049094af0e563a7941c8d609d7635c3ffdcf2cca5c3ce34c3a1fea7572d78f"
Oct 30 05:18:41.239344 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:41.239283    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b049094af0e563a7941c8d609d7635c3ffdcf2cca5c3ce34c3a1fea7572d78f"} err="failed to get container status \"9b049094af0e563a7941c8d609d7635c3ffdcf2cca5c3ce34c3a1fea7572d78f\": rpc error: code = NotFound desc = could not find container \"9b049094af0e563a7941c8d609d7635c3ffdcf2cca5c3ce34c3a1fea7572d78f\": container with ID starting with 9b049094af0e563a7941c8d609d7635c3ffdcf2cca5c3ce34c3a1fea7572d78f not found: ID does not exist"
Oct 30 05:18:41.300493 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:41.300458    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-36fd3c/test-model-36fd3c-kserve-86d45f7b9c-qhhbc"]
Oct 30 05:18:41.304964 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:41.304939    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-36fd3c/test-model-36fd3c-kserve-86d45f7b9c-qhhbc"]
Oct 30 05:18:41.332173 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:41.332144    5898 patch_prober.go:29] interesting pod/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 30 05:18:41.332283 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:41.332190    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w" podUID="8a999b0b-fe75-49de-af7f-bdc3145966c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 30 05:18:41.434473 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:41.434436    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:18:41.533763 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:41.533688    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:18:42.043919 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:42.041404    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:18:42.140079 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:42.140047    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:18:42.240543 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:42.240506    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:18:42.511037 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:42.511000    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w"
Oct 30 05:18:42.512508 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:42.512480    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w"
Oct 30 05:18:42.723887 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:42.722198    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:18:42.895868 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:42.895829    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da336e19-5be4-46ea-80c9-1541cacfa7f0" path="/var/lib/kubelet/pods/da336e19-5be4-46ea-80c9-1541cacfa7f0/volumes"
Oct 30 05:18:43.996332 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:43.996298    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:18:45.742774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:45.742742    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:18:46.970554 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:46.970523    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:18:47.071957 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:47.071923    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:18:47.378649 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:47.378613    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:18:48.699050 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:48.699018    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:18:48.801053 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:48.801017    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:18:49.049059 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:49.048966    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv"]
Oct 30 05:18:49.049492 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:49.049457    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv" podUID="c9bee146-e705-478f-abda-fa3111705a69" containerName="istio-proxy" containerID="cri-o://035d0fbe0bbd1161ec30ac37a257da7946af6bbf1c76e8c83ffd7780ab336135" gracePeriod=30
Oct 30 05:18:49.071100 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:49.071064    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:18:49.180072 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:49.180035    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:18:50.963691 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:50.963633    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:18:52.929492 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:52.929464    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:18:54.347373 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:54.347340    5898 generic.go:356] "Generic (PLEG): container finished" podID="c9bee146-e705-478f-abda-fa3111705a69" containerID="035d0fbe0bbd1161ec30ac37a257da7946af6bbf1c76e8c83ffd7780ab336135" exitCode=0
Oct 30 05:18:54.347806 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:54.347383    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv" event={"ID":"c9bee146-e705-478f-abda-fa3111705a69","Type":"ContainerDied","Data":"035d0fbe0bbd1161ec30ac37a257da7946af6bbf1c76e8c83ffd7780ab336135"}
Oct 30 05:18:54.737274 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:54.737248    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv"
Oct 30 05:18:54.891545 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:54.891464    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9bee146-e705-478f-abda-fa3111705a69-istio-token\" (UniqueName: \"kubernetes.io/projected/c9bee146-e705-478f-abda-fa3111705a69-istio-token\") pod \"c9bee146-e705-478f-abda-fa3111705a69\" (UID: \"c9bee146-e705-478f-abda-fa3111705a69\") "
Oct 30 05:18:54.891723 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:54.891569    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9bee146-e705-478f-abda-fa3111705a69-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9bee146-e705-478f-abda-fa3111705a69-istiod-ca-cert\") pod \"c9bee146-e705-478f-abda-fa3111705a69\" (UID: \"c9bee146-e705-478f-abda-fa3111705a69\") "
Oct 30 05:18:54.891723 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:54.891683    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9bee146-e705-478f-abda-fa3111705a69-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bee146-e705-478f-abda-fa3111705a69-workload-socket\") pod \"c9bee146-e705-478f-abda-fa3111705a69\" (UID: \"c9bee146-e705-478f-abda-fa3111705a69\") "
Oct 30 05:18:54.891972 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:54.891752    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9bee146-e705-478f-abda-fa3111705a69-kube-api-access-659kd\" (UniqueName: \"kubernetes.io/projected/c9bee146-e705-478f-abda-fa3111705a69-kube-api-access-659kd\") pod \"c9bee146-e705-478f-abda-fa3111705a69\" (UID: \"c9bee146-e705-478f-abda-fa3111705a69\") "
Oct 30 05:18:54.891972 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:54.891784    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9bee146-e705-478f-abda-fa3111705a69-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9bee146-e705-478f-abda-fa3111705a69-istio-data\") pod \"c9bee146-e705-478f-abda-fa3111705a69\" (UID: \"c9bee146-e705-478f-abda-fa3111705a69\") "
Oct 30 05:18:54.891972 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:54.891814    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9bee146-e705-478f-abda-fa3111705a69-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9bee146-e705-478f-abda-fa3111705a69-istio-envoy\") pod \"c9bee146-e705-478f-abda-fa3111705a69\" (UID: \"c9bee146-e705-478f-abda-fa3111705a69\") "
Oct 30 05:18:54.891972 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:54.891878    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9bee146-e705-478f-abda-fa3111705a69-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9bee146-e705-478f-abda-fa3111705a69-workload-certs\") pod \"c9bee146-e705-478f-abda-fa3111705a69\" (UID: \"c9bee146-e705-478f-abda-fa3111705a69\") "
Oct 30 05:18:54.892240 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:54.892037    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9bee146-e705-478f-abda-fa3111705a69-workload-socket" pod "c9bee146-e705-478f-abda-fa3111705a69" (UID: "c9bee146-e705-478f-abda-fa3111705a69"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:18:54.892240 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:54.892077    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9bee146-e705-478f-abda-fa3111705a69-istiod-ca-cert" pod "c9bee146-e705-478f-abda-fa3111705a69" (UID: "c9bee146-e705-478f-abda-fa3111705a69"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:18:54.892240 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:54.892222    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9bee146-e705-478f-abda-fa3111705a69-istio-data" pod "c9bee146-e705-478f-abda-fa3111705a69" (UID: "c9bee146-e705-478f-abda-fa3111705a69"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:18:54.892402 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:54.892376    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9bee146-e705-478f-abda-fa3111705a69-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bee146-e705-478f-abda-fa3111705a69-credential-socket\") pod \"c9bee146-e705-478f-abda-fa3111705a69\" (UID: \"c9bee146-e705-478f-abda-fa3111705a69\") "
Oct 30 05:18:54.892457 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:54.892419    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c9bee146-e705-478f-abda-fa3111705a69-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9bee146-e705-478f-abda-fa3111705a69-istio-podinfo\") pod \"c9bee146-e705-478f-abda-fa3111705a69\" (UID: \"c9bee146-e705-478f-abda-fa3111705a69\") "
Oct 30 05:18:54.892972 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:54.892451    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9bee146-e705-478f-abda-fa3111705a69-workload-certs" pod "c9bee146-e705-478f-abda-fa3111705a69" (UID: "c9bee146-e705-478f-abda-fa3111705a69"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:18:54.892972 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:54.892680    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9bee146-e705-478f-abda-fa3111705a69-credential-socket" pod "c9bee146-e705-478f-abda-fa3111705a69" (UID: "c9bee146-e705-478f-abda-fa3111705a69"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:18:54.893266 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:54.893173    5898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9bee146-e705-478f-abda-fa3111705a69-istio-data\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:18:54.893266 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:54.893199    5898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9bee146-e705-478f-abda-fa3111705a69-workload-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:18:54.893266 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:54.893214    5898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bee146-e705-478f-abda-fa3111705a69-credential-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:18:54.893266 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:54.893228    5898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9bee146-e705-478f-abda-fa3111705a69-istiod-ca-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:18:54.893266 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:54.893244    5898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bee146-e705-478f-abda-fa3111705a69-workload-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:18:54.894193 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:54.894165    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9bee146-e705-478f-abda-fa3111705a69-istio-token" pod "c9bee146-e705-478f-abda-fa3111705a69" (UID: "c9bee146-e705-478f-abda-fa3111705a69"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:18:54.894584 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:54.894564    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9bee146-e705-478f-abda-fa3111705a69-kube-api-access-659kd" pod "c9bee146-e705-478f-abda-fa3111705a69" (UID: "c9bee146-e705-478f-abda-fa3111705a69"). InnerVolumeSpecName "kube-api-access-659kd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:18:54.894769 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:54.894750    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9bee146-e705-478f-abda-fa3111705a69-istio-podinfo" pod "c9bee146-e705-478f-abda-fa3111705a69" (UID: "c9bee146-e705-478f-abda-fa3111705a69"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:18:54.895604 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:54.895584    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9bee146-e705-478f-abda-fa3111705a69-istio-envoy" pod "c9bee146-e705-478f-abda-fa3111705a69" (UID: "c9bee146-e705-478f-abda-fa3111705a69"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:18:54.992266 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:54.992229    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:18:54.994843 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:54.994818    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-659kd\" (UniqueName: \"kubernetes.io/projected/c9bee146-e705-478f-abda-fa3111705a69-kube-api-access-659kd\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:18:54.994999 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:54.994848    5898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9bee146-e705-478f-abda-fa3111705a69-istio-envoy\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:18:54.994999 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:54.994867    5898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9bee146-e705-478f-abda-fa3111705a69-istio-podinfo\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:18:54.994999 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:54.994882    5898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9bee146-e705-478f-abda-fa3111705a69-istio-token\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:18:55.356105 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:55.356069    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv"
Oct 30 05:18:55.356578 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:55.356099    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv" event={"ID":"c9bee146-e705-478f-abda-fa3111705a69","Type":"ContainerDied","Data":"135adfdd25dc44d93e850876fcf19e9938f7da98fdf994e1b1bf590adb1829ae"}
Oct 30 05:18:55.356578 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:55.356160    5898 scope.go:122] "RemoveContainer" containerID="035d0fbe0bbd1161ec30ac37a257da7946af6bbf1c76e8c83ffd7780ab336135"
Oct 30 05:18:55.418502 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:55.418474    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv"]
Oct 30 05:18:55.430166 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:55.430139    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-1f8eb4-openshift-default-ccbcf8c75-xdcgv"]
Oct 30 05:18:56.713593 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:56.713563    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:18:56.834561 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:18:56.834522    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:18:56.894728 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:56.894695    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9bee146-e705-478f-abda-fa3111705a69" path="/var/lib/kubelet/pods/c9bee146-e705-478f-abda-fa3111705a69/volumes"
Oct 30 05:18:58.465282 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:58.465250    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:18:58.556988 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:18:58.556879    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:19:00.243191 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:00.243154    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:19:00.328350 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:00.328315    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:19:00.407807 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:00.407770    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:19:02.209539 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:02.209509    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:19:03.674864 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:03.674810    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-010013/test-model-010013-kserve-6bdfbb6f89-9fqsj"]
Oct 30 05:19:03.712452 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:03.712419    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:19:04.057895 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:04.057810    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:19:05.996190 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:05.996151    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:19:06.076811 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:06.076766    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:19:06.310531 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:06.310452    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:19:07.836962 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:07.836930    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:19:09.315601 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:09.315563    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:19:09.405186 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:09.405141    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:19:09.659588 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:09.659542    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:19:11.452678 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:11.452642    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:19:13.054524 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:13.054448    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:19:14.660730 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:14.660694    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:19:16.360136 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:16.360100    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:19:17.248404 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:17.248364    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww"]
Oct 30 05:19:17.249166 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:17.249145    5898 cpu_manager.go:414] "containerMap: removing container" podUID="da336e19-5be4-46ea-80c9-1541cacfa7f0" containerName="main"
Oct 30 05:19:17.249166 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:17.249164    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da336e19-5be4-46ea-80c9-1541cacfa7f0" containerName="main"
Oct 30 05:19:17.249302 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:17.249183    5898 cpu_manager.go:414] "containerMap: removing container" podUID="c9bee146-e705-478f-abda-fa3111705a69" containerName="istio-proxy"
Oct 30 05:19:17.249302 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:17.249189    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9bee146-e705-478f-abda-fa3111705a69" containerName="istio-proxy"
Oct 30 05:19:17.249368 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:17.249334    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9bee146-e705-478f-abda-fa3111705a69" containerName="istio-proxy"
Oct 30 05:19:17.249368 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:17.249354    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="da336e19-5be4-46ea-80c9-1541cacfa7f0" containerName="main"
Oct 30 05:19:17.254495 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:17.254476    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww"
Oct 30 05:19:17.284700 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:17.284670    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww"]
Oct 30 05:19:17.309939 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:17.309862    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d078150b-ed4c-486a-8a9c-762474c26003-istio-token\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww\" (UID: \"d078150b-ed4c-486a-8a9c-762474c26003\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww"
Oct 30 05:19:17.310112 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:17.310042    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d078150b-ed4c-486a-8a9c-762474c26003-workload-certs\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww\" (UID: \"d078150b-ed4c-486a-8a9c-762474c26003\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww"
Oct 30 05:19:17.310112 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:17.310103    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d078150b-ed4c-486a-8a9c-762474c26003-istio-data\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww\" (UID: \"d078150b-ed4c-486a-8a9c-762474c26003\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww"
Oct 30 05:19:17.310236 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:17.310208    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d078150b-ed4c-486a-8a9c-762474c26003-credential-socket\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww\" (UID: \"d078150b-ed4c-486a-8a9c-762474c26003\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww"
Oct 30 05:19:17.310315 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:17.310295    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d078150b-ed4c-486a-8a9c-762474c26003-istio-podinfo\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww\" (UID: \"d078150b-ed4c-486a-8a9c-762474c26003\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww"
Oct 30 05:19:17.310452 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:17.310438    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d078150b-ed4c-486a-8a9c-762474c26003-istiod-ca-cert\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww\" (UID: \"d078150b-ed4c-486a-8a9c-762474c26003\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww"
Oct 30 05:19:17.310501 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:17.310480    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78bxj\" (UniqueName: \"kubernetes.io/projected/d078150b-ed4c-486a-8a9c-762474c26003-kube-api-access-78bxj\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww\" (UID: \"d078150b-ed4c-486a-8a9c-762474c26003\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww"
Oct 30 05:19:17.310559 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:17.310505    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d078150b-ed4c-486a-8a9c-762474c26003-istio-envoy\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww\" (UID: \"d078150b-ed4c-486a-8a9c-762474c26003\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww"
Oct 30 05:19:17.310619 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:17.310603    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d078150b-ed4c-486a-8a9c-762474c26003-workload-socket\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww\" (UID: \"d078150b-ed4c-486a-8a9c-762474c26003\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww"
Oct 30 05:19:17.412042 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:17.412004    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d078150b-ed4c-486a-8a9c-762474c26003-istiod-ca-cert\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww\" (UID: \"d078150b-ed4c-486a-8a9c-762474c26003\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww"
Oct 30 05:19:17.412511 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:17.412058    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78bxj\" (UniqueName: \"kubernetes.io/projected/d078150b-ed4c-486a-8a9c-762474c26003-kube-api-access-78bxj\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww\" (UID: \"d078150b-ed4c-486a-8a9c-762474c26003\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww"
Oct 30 05:19:17.412511 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:17.412083    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d078150b-ed4c-486a-8a9c-762474c26003-istio-envoy\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww\" (UID: \"d078150b-ed4c-486a-8a9c-762474c26003\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww"
Oct 30 05:19:17.412511 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:17.412114    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d078150b-ed4c-486a-8a9c-762474c26003-workload-socket\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww\" (UID: \"d078150b-ed4c-486a-8a9c-762474c26003\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww"
Oct 30 05:19:17.412511 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:17.412150    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d078150b-ed4c-486a-8a9c-762474c26003-istio-token\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww\" (UID: \"d078150b-ed4c-486a-8a9c-762474c26003\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww"
Oct 30 05:19:17.412511 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:17.412226    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d078150b-ed4c-486a-8a9c-762474c26003-workload-certs\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww\" (UID: \"d078150b-ed4c-486a-8a9c-762474c26003\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww"
Oct 30 05:19:17.412511 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:17.412267    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d078150b-ed4c-486a-8a9c-762474c26003-istio-data\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww\" (UID: \"d078150b-ed4c-486a-8a9c-762474c26003\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww"
Oct 30 05:19:17.412511 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:17.412301    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d078150b-ed4c-486a-8a9c-762474c26003-credential-socket\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww\" (UID: \"d078150b-ed4c-486a-8a9c-762474c26003\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww"
Oct 30 05:19:17.412511 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:17.412343    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d078150b-ed4c-486a-8a9c-762474c26003-istio-podinfo\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww\" (UID: \"d078150b-ed4c-486a-8a9c-762474c26003\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww"
Oct 30 05:19:17.412956 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:17.412566    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d078150b-ed4c-486a-8a9c-762474c26003-workload-socket\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww\" (UID: \"d078150b-ed4c-486a-8a9c-762474c26003\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww"
Oct 30 05:19:17.412956 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:17.412612    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d078150b-ed4c-486a-8a9c-762474c26003-workload-certs\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww\" (UID: \"d078150b-ed4c-486a-8a9c-762474c26003\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww"
Oct 30 05:19:17.412956 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:17.412659    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d078150b-ed4c-486a-8a9c-762474c26003-istio-data\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww\" (UID: \"d078150b-ed4c-486a-8a9c-762474c26003\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww"
Oct 30 05:19:17.412956 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:17.412782    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d078150b-ed4c-486a-8a9c-762474c26003-credential-socket\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww\" (UID: \"d078150b-ed4c-486a-8a9c-762474c26003\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww"
Oct 30 05:19:17.412956 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:17.412844    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d078150b-ed4c-486a-8a9c-762474c26003-istiod-ca-cert\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww\" (UID: \"d078150b-ed4c-486a-8a9c-762474c26003\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww"
Oct 30 05:19:17.414946 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:17.414887    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d078150b-ed4c-486a-8a9c-762474c26003-istio-envoy\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww\" (UID: \"d078150b-ed4c-486a-8a9c-762474c26003\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww"
Oct 30 05:19:17.414946 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:17.414893    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d078150b-ed4c-486a-8a9c-762474c26003-istio-podinfo\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww\" (UID: \"d078150b-ed4c-486a-8a9c-762474c26003\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww"
Oct 30 05:19:17.429993 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:17.429968    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78bxj\" (UniqueName: \"kubernetes.io/projected/d078150b-ed4c-486a-8a9c-762474c26003-kube-api-access-78bxj\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww\" (UID: \"d078150b-ed4c-486a-8a9c-762474c26003\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww"
Oct 30 05:19:17.430097 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:17.430027    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d078150b-ed4c-486a-8a9c-762474c26003-istio-token\") pod \"e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww\" (UID: \"d078150b-ed4c-486a-8a9c-762474c26003\") " pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww"
Oct 30 05:19:17.569437 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:17.569366    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww"
Oct 30 05:19:17.891537 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:17.891508    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww"]
Oct 30 05:19:17.896328 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:19:17.896299    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd078150b_ed4c_486a_8a9c_762474c26003.slice/crio-06b37ff3ec42863fa2748e820eb7f0fbcc7f397bdae8e9d2f66679d2dbfca397 WatchSource:0}: Error finding container 06b37ff3ec42863fa2748e820eb7f0fbcc7f397bdae8e9d2f66679d2dbfca397: Status 404 returned error can't find the container with id 06b37ff3ec42863fa2748e820eb7f0fbcc7f397bdae8e9d2f66679d2dbfca397
Oct 30 05:19:17.899218 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:17.899185    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:19:17.899324 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:17.899269    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:19:17.899324 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:17.899314    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:19:18.023967 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:18.023932    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:19:18.537852 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:18.537815    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww" event={"ID":"d078150b-ed4c-486a-8a9c-762474c26003","Type":"ContainerStarted","Data":"fa981ab2b4ca444e4733ef9985c1bbeccbbd97c8c8450d3c02cf9d443f154c13"}
Oct 30 05:19:18.538369 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:18.537880    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww" event={"ID":"d078150b-ed4c-486a-8a9c-762474c26003","Type":"ContainerStarted","Data":"06b37ff3ec42863fa2748e820eb7f0fbcc7f397bdae8e9d2f66679d2dbfca397"}
Oct 30 05:19:18.569871 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:18.569843    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww"
Oct 30 05:19:18.571686 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:18.571659    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 30 05:19:18.571814 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:18.571705    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww" podUID="d078150b-ed4c-486a-8a9c-762474c26003" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 30 05:19:18.592490 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:18.592400    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww" podStartSLOduration=1.5923796000000001 podStartE2EDuration="1.5923796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:19:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:19:18.589018157 +0000 UTC m=+2087.812838076" watchObservedRunningTime="2026-10-30 05:19:18.5923796 +0000 UTC m=+2087.816199505"
Oct 30 05:19:18.707183 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:18.707142    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-010013/test-model-010013-kserve-6bdfbb6f89-9fqsj" podUID="ef140ba8-eb01-40bd-9b95-216c8bacd7af" containerName="main" containerID="cri-o://cf4394eedfdb753f250e79e543329a93e7b403a9cde63821c85f068845e4a26a" gracePeriod=45
Oct 30 05:19:19.055356 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:19.055328    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-010013/test-model-010013-kserve-6bdfbb6f89-9fqsj"
Oct 30 05:19:19.152366 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:19.152295    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef140ba8-eb01-40bd-9b95-216c8bacd7af-kube-api-access-6ddjs\" (UniqueName: \"kubernetes.io/projected/ef140ba8-eb01-40bd-9b95-216c8bacd7af-kube-api-access-6ddjs\") pod \"ef140ba8-eb01-40bd-9b95-216c8bacd7af\" (UID: \"ef140ba8-eb01-40bd-9b95-216c8bacd7af\") "
Oct 30 05:19:19.152366 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:19.152356    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef140ba8-eb01-40bd-9b95-216c8bacd7af-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef140ba8-eb01-40bd-9b95-216c8bacd7af-model-cache\") pod \"ef140ba8-eb01-40bd-9b95-216c8bacd7af\" (UID: \"ef140ba8-eb01-40bd-9b95-216c8bacd7af\") "
Oct 30 05:19:19.152588 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:19.152410    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef140ba8-eb01-40bd-9b95-216c8bacd7af-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef140ba8-eb01-40bd-9b95-216c8bacd7af-dshm\") pod \"ef140ba8-eb01-40bd-9b95-216c8bacd7af\" (UID: \"ef140ba8-eb01-40bd-9b95-216c8bacd7af\") "
Oct 30 05:19:19.152588 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:19.152482    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef140ba8-eb01-40bd-9b95-216c8bacd7af-home\" (UniqueName: \"kubernetes.io/empty-dir/ef140ba8-eb01-40bd-9b95-216c8bacd7af-home\") pod \"ef140ba8-eb01-40bd-9b95-216c8bacd7af\" (UID: \"ef140ba8-eb01-40bd-9b95-216c8bacd7af\") "
Oct 30 05:19:19.152588 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:19.152560    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef140ba8-eb01-40bd-9b95-216c8bacd7af-tls-certs\" (UniqueName: \"kubernetes.io/secret/ef140ba8-eb01-40bd-9b95-216c8bacd7af-tls-certs\") pod \"ef140ba8-eb01-40bd-9b95-216c8bacd7af\" (UID: \"ef140ba8-eb01-40bd-9b95-216c8bacd7af\") "
Oct 30 05:19:19.152822 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:19.152620    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef140ba8-eb01-40bd-9b95-216c8bacd7af-model-cache" pod "ef140ba8-eb01-40bd-9b95-216c8bacd7af" (UID: "ef140ba8-eb01-40bd-9b95-216c8bacd7af"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:19:19.152822 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:19.152661    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef140ba8-eb01-40bd-9b95-216c8bacd7af-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef140ba8-eb01-40bd-9b95-216c8bacd7af-tmp-dir\") pod \"ef140ba8-eb01-40bd-9b95-216c8bacd7af\" (UID: \"ef140ba8-eb01-40bd-9b95-216c8bacd7af\") "
Oct 30 05:19:19.152822 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:19.152750    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef140ba8-eb01-40bd-9b95-216c8bacd7af-home" pod "ef140ba8-eb01-40bd-9b95-216c8bacd7af" (UID: "ef140ba8-eb01-40bd-9b95-216c8bacd7af"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:19:19.152965 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:19.152909    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef140ba8-eb01-40bd-9b95-216c8bacd7af-tmp-dir" pod "ef140ba8-eb01-40bd-9b95-216c8bacd7af" (UID: "ef140ba8-eb01-40bd-9b95-216c8bacd7af"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:19:19.153448 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:19.153425    5898 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef140ba8-eb01-40bd-9b95-216c8bacd7af-model-cache\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:19:19.153448 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:19.153446    5898 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef140ba8-eb01-40bd-9b95-216c8bacd7af-home\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:19:19.153686 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:19.153455    5898 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef140ba8-eb01-40bd-9b95-216c8bacd7af-tmp-dir\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:19:19.154735 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:19.154710    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef140ba8-eb01-40bd-9b95-216c8bacd7af-dshm" pod "ef140ba8-eb01-40bd-9b95-216c8bacd7af" (UID: "ef140ba8-eb01-40bd-9b95-216c8bacd7af"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:19:19.154848 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:19.154825    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef140ba8-eb01-40bd-9b95-216c8bacd7af-kube-api-access-6ddjs" pod "ef140ba8-eb01-40bd-9b95-216c8bacd7af" (UID: "ef140ba8-eb01-40bd-9b95-216c8bacd7af"). InnerVolumeSpecName "kube-api-access-6ddjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:19:19.154986 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:19.154960    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef140ba8-eb01-40bd-9b95-216c8bacd7af-tls-certs" pod "ef140ba8-eb01-40bd-9b95-216c8bacd7af" (UID: "ef140ba8-eb01-40bd-9b95-216c8bacd7af"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:19:19.255049 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:19.255007    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ddjs\" (UniqueName: \"kubernetes.io/projected/ef140ba8-eb01-40bd-9b95-216c8bacd7af-kube-api-access-6ddjs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:19:19.255049 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:19.255043    5898 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef140ba8-eb01-40bd-9b95-216c8bacd7af-dshm\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:19:19.255049 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:19.255058    5898 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef140ba8-eb01-40bd-9b95-216c8bacd7af-tls-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:19:19.546606 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:19.546520    5898 generic.go:356] "Generic (PLEG): container finished" podID="ef140ba8-eb01-40bd-9b95-216c8bacd7af" containerID="cf4394eedfdb753f250e79e543329a93e7b403a9cde63821c85f068845e4a26a" exitCode=0
Oct 30 05:19:19.546606 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:19.546566    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-010013/test-model-010013-kserve-6bdfbb6f89-9fqsj"
Oct 30 05:19:19.547138 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:19.546604    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-010013/test-model-010013-kserve-6bdfbb6f89-9fqsj" event={"ID":"ef140ba8-eb01-40bd-9b95-216c8bacd7af","Type":"ContainerDied","Data":"cf4394eedfdb753f250e79e543329a93e7b403a9cde63821c85f068845e4a26a"}
Oct 30 05:19:19.547138 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:19.546647    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-010013/test-model-010013-kserve-6bdfbb6f89-9fqsj" event={"ID":"ef140ba8-eb01-40bd-9b95-216c8bacd7af","Type":"ContainerDied","Data":"5fa8a5509218bb2848c26b75943a5cbfdb57607a5d833c519e36e009be071608"}
Oct 30 05:19:19.547138 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:19.546663    5898 scope.go:122] "RemoveContainer" containerID="cf4394eedfdb753f250e79e543329a93e7b403a9cde63821c85f068845e4a26a"
Oct 30 05:19:19.557765 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:19.557746    5898 scope.go:122] "RemoveContainer" containerID="cf4394eedfdb753f250e79e543329a93e7b403a9cde63821c85f068845e4a26a"
Oct 30 05:19:19.558037 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:19:19.558020    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf4394eedfdb753f250e79e543329a93e7b403a9cde63821c85f068845e4a26a\": container with ID starting with cf4394eedfdb753f250e79e543329a93e7b403a9cde63821c85f068845e4a26a not found: ID does not exist" containerID="cf4394eedfdb753f250e79e543329a93e7b403a9cde63821c85f068845e4a26a"
Oct 30 05:19:19.558090 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:19.558044    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf4394eedfdb753f250e79e543329a93e7b403a9cde63821c85f068845e4a26a"} err="failed to get container status \"cf4394eedfdb753f250e79e543329a93e7b403a9cde63821c85f068845e4a26a\": rpc error: code = NotFound desc = could not find container \"cf4394eedfdb753f250e79e543329a93e7b403a9cde63821c85f068845e4a26a\": container with ID starting with cf4394eedfdb753f250e79e543329a93e7b403a9cde63821c85f068845e4a26a not found: ID does not exist"
Oct 30 05:19:19.569951 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:19.569926    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 30 05:19:19.570039 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:19.569971    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww" podUID="d078150b-ed4c-486a-8a9c-762474c26003" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 30 05:19:19.603077 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:19.603036    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-010013/test-model-010013-kserve-6bdfbb6f89-9fqsj"]
Oct 30 05:19:19.610447 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:19.610417    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-010013/test-model-010013-kserve-6bdfbb6f89-9fqsj"]
Oct 30 05:19:19.841618 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:19.841586    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:19:20.570741 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:20.570706    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 30 05:19:20.571154 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:20.570773    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww" podUID="d078150b-ed4c-486a-8a9c-762474c26003" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 30 05:19:20.910066 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:20.910025    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef140ba8-eb01-40bd-9b95-216c8bacd7af" path="/var/lib/kubelet/pods/ef140ba8-eb01-40bd-9b95-216c8bacd7af/volumes"
Oct 30 05:19:21.569920 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:21.569857    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 30 05:19:21.570100 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:21.569942    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww" podUID="d078150b-ed4c-486a-8a9c-762474c26003" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 30 05:19:21.792514 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:21.792474    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:19:22.569651 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:22.569614    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 30 05:19:22.569858 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:22.569676    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww" podUID="d078150b-ed4c-486a-8a9c-762474c26003" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 30 05:19:23.552761 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:23.552721    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:19:23.570342 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:23.570316    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 30 05:19:23.570487 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:23.570362    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww" podUID="d078150b-ed4c-486a-8a9c-762474c26003" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 30 05:19:24.575779 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:24.575749    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww"
Oct 30 05:19:24.576252 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:24.576204    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww"
Oct 30 05:19:24.576989 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:24.576968    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww"
Oct 30 05:19:25.276800 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:25.276759    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:19:25.703500 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:25.703468    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:19:26.819192 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:19:26.819156    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:19:27.166921 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:27.164979    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:19:29.057006 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:29.056970    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:19:30.823769 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:30.823727    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:19:32.771946 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:32.771884    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:19:33.639917 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:33.639587    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-8afb26/test-model-8afb26-kserve-6c6f6d8777-7mlmd"]
Oct 30 05:19:33.686936 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:33.686637    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:19:34.879050 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:34.879014    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:19:35.353873 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:35.353829    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:19:36.721691 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:36.721653    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:19:38.137038 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:38.136994    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:19:38.750583 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:38.750545    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:19:39.048964 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:39.048867    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:19:39.205372 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:39.205333    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:19:39.605978 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:39.605949    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:19:41.214194 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:41.214163    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf"]
Oct 30 05:19:41.214600 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:41.214373    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf" podUID="b5d063ac-6da8-4217-bddd-82a05db7bc50" containerName="istio-proxy" containerID="cri-o://48e96e61aa3964057d215d81111d0e267b88a3bbafcd91aeef75ef3fb8105a71" gracePeriod=30
Oct 30 05:19:41.243453 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:41.243407    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:19:41.548477 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:41.548399    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2"]
Oct 30 05:19:41.548640 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:41.548596    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2" podUID="983a192c-67a4-4684-86fc-6df70f38a1ed" containerName="istio-proxy" containerID="cri-o://490d7a49f4d76a3088a22e1fbc1a4ff8b3584250f52ebe3cb712e4b293542202" gracePeriod=30
Oct 30 05:19:41.617850 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:41.617804    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:19:42.947622 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:42.947521    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:19:44.494373 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:44.494336    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:19:44.602935 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:44.600728    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:19:44.695505 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:44.695453    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:19:45.459080 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:45.459046    5898 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 05:19:45.459252 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:45.459104    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf" podUID="b5d063ac-6da8-4217-bddd-82a05db7bc50" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 05:19:45.608892 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:45.608858    5898 patch_prober.go:29] interesting pod/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 05:19:45.609360 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:45.608954    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2" podUID="983a192c-67a4-4684-86fc-6df70f38a1ed" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 05:19:46.035963 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:46.035918    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89"]
Oct 30 05:19:46.036163 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:46.036137    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89" podUID="01b63947-ab2c-4052-91a9-f91a1a499168" containerName="istio-proxy" containerID="cri-o://b27bd1182ec2712d85a1e46a896fc322e493c44280df6cda6df4c6a59fab14d4" gracePeriod=30
Oct 30 05:19:46.277182 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:46.277153    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:19:46.377883 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:46.377840    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:19:46.530044 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:46.530016    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf"
Oct 30 05:19:46.641370 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:46.641332    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw"]
Oct 30 05:19:46.641877 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:46.641576    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw" podUID="8aed7491-445f-4c4a-86c1-3c844692deee" containerName="istio-proxy" containerID="cri-o://54f3d0f249e07023ad63a3e0495ab1f3504347e51b5c2a8e4f0067dad6268861" gracePeriod=30
Oct 30 05:19:46.656992 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:46.656154    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:19:46.686481 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:46.686448    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5d063ac-6da8-4217-bddd-82a05db7bc50-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5d063ac-6da8-4217-bddd-82a05db7bc50-istio-data\") pod \"b5d063ac-6da8-4217-bddd-82a05db7bc50\" (UID: \"b5d063ac-6da8-4217-bddd-82a05db7bc50\") "
Oct 30 05:19:46.686653 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:46.686525    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5d063ac-6da8-4217-bddd-82a05db7bc50-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5d063ac-6da8-4217-bddd-82a05db7bc50-workload-certs\") pod \"b5d063ac-6da8-4217-bddd-82a05db7bc50\" (UID: \"b5d063ac-6da8-4217-bddd-82a05db7bc50\") "
Oct 30 05:19:46.686653 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:46.686560    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b5d063ac-6da8-4217-bddd-82a05db7bc50-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5d063ac-6da8-4217-bddd-82a05db7bc50-istio-podinfo\") pod \"b5d063ac-6da8-4217-bddd-82a05db7bc50\" (UID: \"b5d063ac-6da8-4217-bddd-82a05db7bc50\") "
Oct 30 05:19:46.686813 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:46.686671    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5d063ac-6da8-4217-bddd-82a05db7bc50-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5d063ac-6da8-4217-bddd-82a05db7bc50-istio-envoy\") pod \"b5d063ac-6da8-4217-bddd-82a05db7bc50\" (UID: \"b5d063ac-6da8-4217-bddd-82a05db7bc50\") "
Oct 30 05:19:46.686813 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:46.686714    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5d063ac-6da8-4217-bddd-82a05db7bc50-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5d063ac-6da8-4217-bddd-82a05db7bc50-istiod-ca-cert\") pod \"b5d063ac-6da8-4217-bddd-82a05db7bc50\" (UID: \"b5d063ac-6da8-4217-bddd-82a05db7bc50\") "
Oct 30 05:19:46.686813 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:46.686717    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5d063ac-6da8-4217-bddd-82a05db7bc50-istio-data" pod "b5d063ac-6da8-4217-bddd-82a05db7bc50" (UID: "b5d063ac-6da8-4217-bddd-82a05db7bc50"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:19:46.687000 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:46.686812    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5d063ac-6da8-4217-bddd-82a05db7bc50-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d063ac-6da8-4217-bddd-82a05db7bc50-credential-socket\") pod \"b5d063ac-6da8-4217-bddd-82a05db7bc50\" (UID: \"b5d063ac-6da8-4217-bddd-82a05db7bc50\") "
Oct 30 05:19:46.687000 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:46.686822    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5d063ac-6da8-4217-bddd-82a05db7bc50-workload-certs" pod "b5d063ac-6da8-4217-bddd-82a05db7bc50" (UID: "b5d063ac-6da8-4217-bddd-82a05db7bc50"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:19:46.687000 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:46.686960    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5d063ac-6da8-4217-bddd-82a05db7bc50-kube-api-access-7l6v6\" (UniqueName: \"kubernetes.io/projected/b5d063ac-6da8-4217-bddd-82a05db7bc50-kube-api-access-7l6v6\") pod \"b5d063ac-6da8-4217-bddd-82a05db7bc50\" (UID: \"b5d063ac-6da8-4217-bddd-82a05db7bc50\") "
Oct 30 05:19:46.687000 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:46.686994    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5d063ac-6da8-4217-bddd-82a05db7bc50-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d063ac-6da8-4217-bddd-82a05db7bc50-workload-socket\") pod \"b5d063ac-6da8-4217-bddd-82a05db7bc50\" (UID: \"b5d063ac-6da8-4217-bddd-82a05db7bc50\") "
Oct 30 05:19:46.687229 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:46.687101    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5d063ac-6da8-4217-bddd-82a05db7bc50-istio-token\" (UniqueName: \"kubernetes.io/projected/b5d063ac-6da8-4217-bddd-82a05db7bc50-istio-token\") pod \"b5d063ac-6da8-4217-bddd-82a05db7bc50\" (UID: \"b5d063ac-6da8-4217-bddd-82a05db7bc50\") "
Oct 30 05:19:46.687229 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:46.687127    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5d063ac-6da8-4217-bddd-82a05db7bc50-credential-socket" pod "b5d063ac-6da8-4217-bddd-82a05db7bc50" (UID: "b5d063ac-6da8-4217-bddd-82a05db7bc50"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:19:46.687229 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:46.687135    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5d063ac-6da8-4217-bddd-82a05db7bc50-istiod-ca-cert" pod "b5d063ac-6da8-4217-bddd-82a05db7bc50" (UID: "b5d063ac-6da8-4217-bddd-82a05db7bc50"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:19:46.687405 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:46.687381    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5d063ac-6da8-4217-bddd-82a05db7bc50-workload-socket" pod "b5d063ac-6da8-4217-bddd-82a05db7bc50" (UID: "b5d063ac-6da8-4217-bddd-82a05db7bc50"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:19:46.687862 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:46.687838    5898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5d063ac-6da8-4217-bddd-82a05db7bc50-istio-data\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:19:46.688216 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:46.687867    5898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5d063ac-6da8-4217-bddd-82a05db7bc50-workload-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:19:46.688216 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:46.687880    5898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5d063ac-6da8-4217-bddd-82a05db7bc50-istiod-ca-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:19:46.688216 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:46.687894    5898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d063ac-6da8-4217-bddd-82a05db7bc50-credential-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:19:46.688216 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:46.687920    5898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d063ac-6da8-4217-bddd-82a05db7bc50-workload-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:19:46.689615 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:46.689581    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5d063ac-6da8-4217-bddd-82a05db7bc50-kube-api-access-7l6v6" pod "b5d063ac-6da8-4217-bddd-82a05db7bc50" (UID: "b5d063ac-6da8-4217-bddd-82a05db7bc50"). InnerVolumeSpecName "kube-api-access-7l6v6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:19:46.690469 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:46.690432    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b5d063ac-6da8-4217-bddd-82a05db7bc50-istio-podinfo" pod "b5d063ac-6da8-4217-bddd-82a05db7bc50" (UID: "b5d063ac-6da8-4217-bddd-82a05db7bc50"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:19:46.690633 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:46.690609    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5d063ac-6da8-4217-bddd-82a05db7bc50-istio-envoy" pod "b5d063ac-6da8-4217-bddd-82a05db7bc50" (UID: "b5d063ac-6da8-4217-bddd-82a05db7bc50"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:19:46.690764 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:46.690743    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5d063ac-6da8-4217-bddd-82a05db7bc50-istio-token" pod "b5d063ac-6da8-4217-bddd-82a05db7bc50" (UID: "b5d063ac-6da8-4217-bddd-82a05db7bc50"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:19:46.825432 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:46.822353    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:19:46.825432 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:46.824003    5898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5d063ac-6da8-4217-bddd-82a05db7bc50-istio-envoy\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:19:46.825432 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:46.824029    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l6v6\" (UniqueName: \"kubernetes.io/projected/b5d063ac-6da8-4217-bddd-82a05db7bc50-kube-api-access-7l6v6\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:19:46.825432 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:46.824045    5898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5d063ac-6da8-4217-bddd-82a05db7bc50-istio-token\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:19:46.825432 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:46.824060    5898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5d063ac-6da8-4217-bddd-82a05db7bc50-istio-podinfo\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:19:46.989438 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:46.989343    5898 generic.go:356] "Generic (PLEG): container finished" podID="b5d063ac-6da8-4217-bddd-82a05db7bc50" containerID="48e96e61aa3964057d215d81111d0e267b88a3bbafcd91aeef75ef3fb8105a71" exitCode=0
Oct 30 05:19:46.989633 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:46.989535    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf"
Oct 30 05:19:46.991051 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:46.991018    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf" event={"ID":"b5d063ac-6da8-4217-bddd-82a05db7bc50","Type":"ContainerDied","Data":"48e96e61aa3964057d215d81111d0e267b88a3bbafcd91aeef75ef3fb8105a71"}
Oct 30 05:19:46.991178 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:46.991077    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf" event={"ID":"b5d063ac-6da8-4217-bddd-82a05db7bc50","Type":"ContainerDied","Data":"e51b3f4f591cf82a1daef302807e5eb76b78fef0ed16bceadb80a1441792e65b"}
Oct 30 05:19:46.991178 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:46.991109    5898 scope.go:122] "RemoveContainer" containerID="48e96e61aa3964057d215d81111d0e267b88a3bbafcd91aeef75ef3fb8105a71"
Oct 30 05:19:46.994270 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:46.994207    5898 generic.go:356] "Generic (PLEG): container finished" podID="983a192c-67a4-4684-86fc-6df70f38a1ed" containerID="490d7a49f4d76a3088a22e1fbc1a4ff8b3584250f52ebe3cb712e4b293542202" exitCode=0
Oct 30 05:19:46.994537 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:46.994517    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2" event={"ID":"983a192c-67a4-4684-86fc-6df70f38a1ed","Type":"ContainerDied","Data":"490d7a49f4d76a3088a22e1fbc1a4ff8b3584250f52ebe3cb712e4b293542202"}
Oct 30 05:19:47.010434 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:47.008532    5898 scope.go:122] "RemoveContainer" containerID="48e96e61aa3964057d215d81111d0e267b88a3bbafcd91aeef75ef3fb8105a71"
Oct 30 05:19:47.010434 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:19:47.009057    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48e96e61aa3964057d215d81111d0e267b88a3bbafcd91aeef75ef3fb8105a71\": container with ID starting with 48e96e61aa3964057d215d81111d0e267b88a3bbafcd91aeef75ef3fb8105a71 not found: ID does not exist" containerID="48e96e61aa3964057d215d81111d0e267b88a3bbafcd91aeef75ef3fb8105a71"
Oct 30 05:19:47.010434 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:47.009091    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48e96e61aa3964057d215d81111d0e267b88a3bbafcd91aeef75ef3fb8105a71"} err="failed to get container status \"48e96e61aa3964057d215d81111d0e267b88a3bbafcd91aeef75ef3fb8105a71\": rpc error: code = NotFound desc = could not find container \"48e96e61aa3964057d215d81111d0e267b88a3bbafcd91aeef75ef3fb8105a71\": container with ID starting with 48e96e61aa3964057d215d81111d0e267b88a3bbafcd91aeef75ef3fb8105a71 not found: ID does not exist"
Oct 30 05:19:47.022974 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:47.022953    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2"
Oct 30 05:19:47.054928 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:47.054886    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf"]
Oct 30 05:19:47.072326 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:47.072300    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-cgncf"]
Oct 30 05:19:47.132665 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:47.132637    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/983a192c-67a4-4684-86fc-6df70f38a1ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/983a192c-67a4-4684-86fc-6df70f38a1ed-credential-socket\") pod \"983a192c-67a4-4684-86fc-6df70f38a1ed\" (UID: \"983a192c-67a4-4684-86fc-6df70f38a1ed\") "
Oct 30 05:19:47.132807 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:47.132708    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/983a192c-67a4-4684-86fc-6df70f38a1ed-kube-api-access-wq96x\" (UniqueName: \"kubernetes.io/projected/983a192c-67a4-4684-86fc-6df70f38a1ed-kube-api-access-wq96x\") pod \"983a192c-67a4-4684-86fc-6df70f38a1ed\" (UID: \"983a192c-67a4-4684-86fc-6df70f38a1ed\") "
Oct 30 05:19:47.132869 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:47.132833    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/983a192c-67a4-4684-86fc-6df70f38a1ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/983a192c-67a4-4684-86fc-6df70f38a1ed-istio-podinfo\") pod \"983a192c-67a4-4684-86fc-6df70f38a1ed\" (UID: \"983a192c-67a4-4684-86fc-6df70f38a1ed\") "
Oct 30 05:19:47.132869 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:47.132858    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/983a192c-67a4-4684-86fc-6df70f38a1ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/983a192c-67a4-4684-86fc-6df70f38a1ed-istio-data\") pod \"983a192c-67a4-4684-86fc-6df70f38a1ed\" (UID: \"983a192c-67a4-4684-86fc-6df70f38a1ed\") "
Oct 30 05:19:47.133008 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:47.132932    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/983a192c-67a4-4684-86fc-6df70f38a1ed-credential-socket" pod "983a192c-67a4-4684-86fc-6df70f38a1ed" (UID: "983a192c-67a4-4684-86fc-6df70f38a1ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:19:47.133008 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:47.132946    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/983a192c-67a4-4684-86fc-6df70f38a1ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/983a192c-67a4-4684-86fc-6df70f38a1ed-workload-socket\") pod \"983a192c-67a4-4684-86fc-6df70f38a1ed\" (UID: \"983a192c-67a4-4684-86fc-6df70f38a1ed\") "
Oct 30 05:19:47.133008 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:47.132982    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/983a192c-67a4-4684-86fc-6df70f38a1ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/983a192c-67a4-4684-86fc-6df70f38a1ed-istiod-ca-cert\") pod \"983a192c-67a4-4684-86fc-6df70f38a1ed\" (UID: \"983a192c-67a4-4684-86fc-6df70f38a1ed\") "
Oct 30 05:19:47.133166 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:47.133041    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/983a192c-67a4-4684-86fc-6df70f38a1ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/983a192c-67a4-4684-86fc-6df70f38a1ed-istio-envoy\") pod \"983a192c-67a4-4684-86fc-6df70f38a1ed\" (UID: \"983a192c-67a4-4684-86fc-6df70f38a1ed\") "
Oct 30 05:19:47.133166 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:47.133068    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/983a192c-67a4-4684-86fc-6df70f38a1ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/983a192c-67a4-4684-86fc-6df70f38a1ed-workload-certs\") pod \"983a192c-67a4-4684-86fc-6df70f38a1ed\" (UID: \"983a192c-67a4-4684-86fc-6df70f38a1ed\") "
Oct 30 05:19:47.133166 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:47.133124    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/983a192c-67a4-4684-86fc-6df70f38a1ed-istio-token\" (UniqueName: \"kubernetes.io/projected/983a192c-67a4-4684-86fc-6df70f38a1ed-istio-token\") pod \"983a192c-67a4-4684-86fc-6df70f38a1ed\" (UID: \"983a192c-67a4-4684-86fc-6df70f38a1ed\") "
Oct 30 05:19:47.133321 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:47.133180    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/983a192c-67a4-4684-86fc-6df70f38a1ed-istio-data" pod "983a192c-67a4-4684-86fc-6df70f38a1ed" (UID: "983a192c-67a4-4684-86fc-6df70f38a1ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:19:47.133321 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:47.133223    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/983a192c-67a4-4684-86fc-6df70f38a1ed-workload-socket" pod "983a192c-67a4-4684-86fc-6df70f38a1ed" (UID: "983a192c-67a4-4684-86fc-6df70f38a1ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:19:47.133598 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:47.133576    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/983a192c-67a4-4684-86fc-6df70f38a1ed-workload-certs" pod "983a192c-67a4-4684-86fc-6df70f38a1ed" (UID: "983a192c-67a4-4684-86fc-6df70f38a1ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:19:47.133938 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:47.133918    5898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/983a192c-67a4-4684-86fc-6df70f38a1ed-istio-data\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:19:47.134569 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:47.133944    5898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/983a192c-67a4-4684-86fc-6df70f38a1ed-workload-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:19:47.134569 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:47.133951    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/983a192c-67a4-4684-86fc-6df70f38a1ed-istiod-ca-cert" pod "983a192c-67a4-4684-86fc-6df70f38a1ed" (UID: "983a192c-67a4-4684-86fc-6df70f38a1ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:19:47.134569 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:47.133962    5898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/983a192c-67a4-4684-86fc-6df70f38a1ed-workload-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:19:47.134569 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:47.133997    5898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/983a192c-67a4-4684-86fc-6df70f38a1ed-credential-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:19:47.135228 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:47.135200    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/983a192c-67a4-4684-86fc-6df70f38a1ed-kube-api-access-wq96x" pod "983a192c-67a4-4684-86fc-6df70f38a1ed" (UID: "983a192c-67a4-4684-86fc-6df70f38a1ed"). InnerVolumeSpecName "kube-api-access-wq96x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:19:47.135364 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:47.135348    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/983a192c-67a4-4684-86fc-6df70f38a1ed-istio-token" pod "983a192c-67a4-4684-86fc-6df70f38a1ed" (UID: "983a192c-67a4-4684-86fc-6df70f38a1ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:19:47.135544 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:47.135513    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/983a192c-67a4-4684-86fc-6df70f38a1ed-istio-podinfo" pod "983a192c-67a4-4684-86fc-6df70f38a1ed" (UID: "983a192c-67a4-4684-86fc-6df70f38a1ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:19:47.135778 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:47.135760    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/983a192c-67a4-4684-86fc-6df70f38a1ed-istio-envoy" pod "983a192c-67a4-4684-86fc-6df70f38a1ed" (UID: "983a192c-67a4-4684-86fc-6df70f38a1ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:19:47.235574 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:47.235535    5898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/983a192c-67a4-4684-86fc-6df70f38a1ed-istiod-ca-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:19:47.235574 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:47.235572    5898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/983a192c-67a4-4684-86fc-6df70f38a1ed-istio-envoy\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:19:47.235574 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:47.235581    5898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/983a192c-67a4-4684-86fc-6df70f38a1ed-istio-token\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:19:47.235804 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:47.235589    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wq96x\" (UniqueName: \"kubernetes.io/projected/983a192c-67a4-4684-86fc-6df70f38a1ed-kube-api-access-wq96x\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:19:47.235804 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:47.235599    5898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/983a192c-67a4-4684-86fc-6df70f38a1ed-istio-podinfo\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:19:47.554007 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:47.553862    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:19:48.010564 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:48.007544    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2"
Oct 30 05:19:48.010564 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:48.007547    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2" event={"ID":"983a192c-67a4-4684-86fc-6df70f38a1ed","Type":"ContainerDied","Data":"4a6323685f3b09c1e0bbc9f253d767e484d8701eca5105ade44d9153c878a668"}
Oct 30 05:19:48.010564 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:48.007979    5898 scope.go:122] "RemoveContainer" containerID="490d7a49f4d76a3088a22e1fbc1a4ff8b3584250f52ebe3cb712e4b293542202"
Oct 30 05:19:48.025253 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:48.025221    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:19:48.074560 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:48.074524    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2"]
Oct 30 05:19:48.085860 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:48.085833    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-v7md2"]
Oct 30 05:19:48.158950 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:48.158915    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:19:48.670039 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:48.669993    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-8afb26/test-model-8afb26-kserve-6c6f6d8777-7mlmd" podUID="fac1e366-ef3a-45e5-bd96-8ebf290f0b5f" containerName="main" containerID="cri-o://9cca65e2ccf236be9c73420b9d502eb88c8c55925e71e785df0db6fc69c53e9a" gracePeriod=45
Oct 30 05:19:48.726589 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:48.726557    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:19:48.894440 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:48.894410    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="983a192c-67a4-4684-86fc-6df70f38a1ed" path="/var/lib/kubelet/pods/983a192c-67a4-4684-86fc-6df70f38a1ed/volumes"
Oct 30 05:19:48.895132 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:48.895079    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5d063ac-6da8-4217-bddd-82a05db7bc50" path="/var/lib/kubelet/pods/b5d063ac-6da8-4217-bddd-82a05db7bc50/volumes"
Oct 30 05:19:49.010488 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:49.010455    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8afb26/test-model-8afb26-kserve-6c6f6d8777-7mlmd"
Oct 30 05:19:49.019438 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:49.019409    5898 generic.go:356] "Generic (PLEG): container finished" podID="fac1e366-ef3a-45e5-bd96-8ebf290f0b5f" containerID="9cca65e2ccf236be9c73420b9d502eb88c8c55925e71e785df0db6fc69c53e9a" exitCode=0
Oct 30 05:19:49.019784 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:49.019685    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8afb26/test-model-8afb26-kserve-6c6f6d8777-7mlmd"
Oct 30 05:19:49.019854 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:49.019834    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8afb26/test-model-8afb26-kserve-6c6f6d8777-7mlmd" event={"ID":"fac1e366-ef3a-45e5-bd96-8ebf290f0b5f","Type":"ContainerDied","Data":"9cca65e2ccf236be9c73420b9d502eb88c8c55925e71e785df0db6fc69c53e9a"}
Oct 30 05:19:49.019889 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:49.019873    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8afb26/test-model-8afb26-kserve-6c6f6d8777-7mlmd" event={"ID":"fac1e366-ef3a-45e5-bd96-8ebf290f0b5f","Type":"ContainerDied","Data":"8736229175be6e7d0344843dd6f5ee89acde16d742e2a2d4fa50c52c0a012868"}
Oct 30 05:19:49.019976 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:49.019895    5898 scope.go:122] "RemoveContainer" containerID="9cca65e2ccf236be9c73420b9d502eb88c8c55925e71e785df0db6fc69c53e9a"
Oct 30 05:19:49.031358 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:49.031337    5898 scope.go:122] "RemoveContainer" containerID="9cca65e2ccf236be9c73420b9d502eb88c8c55925e71e785df0db6fc69c53e9a"
Oct 30 05:19:49.031657 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:19:49.031638    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cca65e2ccf236be9c73420b9d502eb88c8c55925e71e785df0db6fc69c53e9a\": container with ID starting with 9cca65e2ccf236be9c73420b9d502eb88c8c55925e71e785df0db6fc69c53e9a not found: ID does not exist" containerID="9cca65e2ccf236be9c73420b9d502eb88c8c55925e71e785df0db6fc69c53e9a"
Oct 30 05:19:49.031712 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:49.031665    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cca65e2ccf236be9c73420b9d502eb88c8c55925e71e785df0db6fc69c53e9a"} err="failed to get container status \"9cca65e2ccf236be9c73420b9d502eb88c8c55925e71e785df0db6fc69c53e9a\": rpc error: code = NotFound desc = could not find container \"9cca65e2ccf236be9c73420b9d502eb88c8c55925e71e785df0db6fc69c53e9a\": container with ID starting with 9cca65e2ccf236be9c73420b9d502eb88c8c55925e71e785df0db6fc69c53e9a not found: ID does not exist"
Oct 30 05:19:49.076310 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:49.076277    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fac1e366-ef3a-45e5-bd96-8ebf290f0b5f-home\" (UniqueName: \"kubernetes.io/empty-dir/fac1e366-ef3a-45e5-bd96-8ebf290f0b5f-home\") pod \"fac1e366-ef3a-45e5-bd96-8ebf290f0b5f\" (UID: \"fac1e366-ef3a-45e5-bd96-8ebf290f0b5f\") "
Oct 30 05:19:49.076495 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:49.076326    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fac1e366-ef3a-45e5-bd96-8ebf290f0b5f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fac1e366-ef3a-45e5-bd96-8ebf290f0b5f-dshm\") pod \"fac1e366-ef3a-45e5-bd96-8ebf290f0b5f\" (UID: \"fac1e366-ef3a-45e5-bd96-8ebf290f0b5f\") "
Oct 30 05:19:49.076495 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:49.076387    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fac1e366-ef3a-45e5-bd96-8ebf290f0b5f-tls-certs\" (UniqueName: \"kubernetes.io/secret/fac1e366-ef3a-45e5-bd96-8ebf290f0b5f-tls-certs\") pod \"fac1e366-ef3a-45e5-bd96-8ebf290f0b5f\" (UID: \"fac1e366-ef3a-45e5-bd96-8ebf290f0b5f\") "
Oct 30 05:19:49.076627 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:49.076500    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fac1e366-ef3a-45e5-bd96-8ebf290f0b5f-kube-api-access-2x7dk\" (UniqueName: \"kubernetes.io/projected/fac1e366-ef3a-45e5-bd96-8ebf290f0b5f-kube-api-access-2x7dk\") pod \"fac1e366-ef3a-45e5-bd96-8ebf290f0b5f\" (UID: \"fac1e366-ef3a-45e5-bd96-8ebf290f0b5f\") "
Oct 30 05:19:49.076687 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:49.076626    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fac1e366-ef3a-45e5-bd96-8ebf290f0b5f-home" pod "fac1e366-ef3a-45e5-bd96-8ebf290f0b5f" (UID: "fac1e366-ef3a-45e5-bd96-8ebf290f0b5f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:19:49.076687 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:49.076646    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fac1e366-ef3a-45e5-bd96-8ebf290f0b5f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fac1e366-ef3a-45e5-bd96-8ebf290f0b5f-tmp-dir\") pod \"fac1e366-ef3a-45e5-bd96-8ebf290f0b5f\" (UID: \"fac1e366-ef3a-45e5-bd96-8ebf290f0b5f\") "
Oct 30 05:19:49.076796 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:49.076737    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fac1e366-ef3a-45e5-bd96-8ebf290f0b5f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fac1e366-ef3a-45e5-bd96-8ebf290f0b5f-model-cache\") pod \"fac1e366-ef3a-45e5-bd96-8ebf290f0b5f\" (UID: \"fac1e366-ef3a-45e5-bd96-8ebf290f0b5f\") "
Oct 30 05:19:49.077445 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:49.077381    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fac1e366-ef3a-45e5-bd96-8ebf290f0b5f-tmp-dir" pod "fac1e366-ef3a-45e5-bd96-8ebf290f0b5f" (UID: "fac1e366-ef3a-45e5-bd96-8ebf290f0b5f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:19:49.077560 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:49.077491    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fac1e366-ef3a-45e5-bd96-8ebf290f0b5f-model-cache" pod "fac1e366-ef3a-45e5-bd96-8ebf290f0b5f" (UID: "fac1e366-ef3a-45e5-bd96-8ebf290f0b5f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:19:49.078021 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:49.077998    5898 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fac1e366-ef3a-45e5-bd96-8ebf290f0b5f-home\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:19:49.078135 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:49.078026    5898 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fac1e366-ef3a-45e5-bd96-8ebf290f0b5f-tmp-dir\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:19:49.078135 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:49.078041    5898 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fac1e366-ef3a-45e5-bd96-8ebf290f0b5f-model-cache\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:19:49.081717 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:49.081695    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fac1e366-ef3a-45e5-bd96-8ebf290f0b5f-dshm" pod "fac1e366-ef3a-45e5-bd96-8ebf290f0b5f" (UID: "fac1e366-ef3a-45e5-bd96-8ebf290f0b5f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:19:49.082049 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:49.082027    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fac1e366-ef3a-45e5-bd96-8ebf290f0b5f-kube-api-access-2x7dk" pod "fac1e366-ef3a-45e5-bd96-8ebf290f0b5f" (UID: "fac1e366-ef3a-45e5-bd96-8ebf290f0b5f"). InnerVolumeSpecName "kube-api-access-2x7dk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:19:49.083035 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:49.083010    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fac1e366-ef3a-45e5-bd96-8ebf290f0b5f-tls-certs" pod "fac1e366-ef3a-45e5-bd96-8ebf290f0b5f" (UID: "fac1e366-ef3a-45e5-bd96-8ebf290f0b5f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:19:49.179816 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:49.179777    5898 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fac1e366-ef3a-45e5-bd96-8ebf290f0b5f-dshm\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:19:49.179816 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:49.179806    5898 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fac1e366-ef3a-45e5-bd96-8ebf290f0b5f-tls-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:19:49.179816 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:49.179817    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2x7dk\" (UniqueName: \"kubernetes.io/projected/fac1e366-ef3a-45e5-bd96-8ebf290f0b5f-kube-api-access-2x7dk\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:19:49.368599 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:49.368565    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-8afb26/test-model-8afb26-kserve-6c6f6d8777-7mlmd"]
Oct 30 05:19:49.379092 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:49.379064    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-8afb26/test-model-8afb26-kserve-6c6f6d8777-7mlmd"]
Oct 30 05:19:49.759955 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:49.759096    5898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 05:19:49.759955 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:49.759176    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89" podUID="01b63947-ab2c-4052-91a9-f91a1a499168" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 05:19:50.340777 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:50.340735    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:19:50.895444 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:50.895411    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fac1e366-ef3a-45e5-bd96-8ebf290f0b5f" path="/var/lib/kubelet/pods/fac1e366-ef3a-45e5-bd96-8ebf290f0b5f/volumes"
Oct 30 05:19:51.330174 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:51.330147    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89"
Oct 30 05:19:51.425131 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:51.424618    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/01b63947-ab2c-4052-91a9-f91a1a499168-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01b63947-ab2c-4052-91a9-f91a1a499168-istio-podinfo\") pod \"01b63947-ab2c-4052-91a9-f91a1a499168\" (UID: \"01b63947-ab2c-4052-91a9-f91a1a499168\") "
Oct 30 05:19:51.425807 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:51.425780    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01b63947-ab2c-4052-91a9-f91a1a499168-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01b63947-ab2c-4052-91a9-f91a1a499168-istio-envoy\") pod \"01b63947-ab2c-4052-91a9-f91a1a499168\" (UID: \"01b63947-ab2c-4052-91a9-f91a1a499168\") "
Oct 30 05:19:51.426091 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:51.426069    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01b63947-ab2c-4052-91a9-f91a1a499168-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01b63947-ab2c-4052-91a9-f91a1a499168-workload-certs\") pod \"01b63947-ab2c-4052-91a9-f91a1a499168\" (UID: \"01b63947-ab2c-4052-91a9-f91a1a499168\") "
Oct 30 05:19:51.426265 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:51.426250    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01b63947-ab2c-4052-91a9-f91a1a499168-istio-token\" (UniqueName: \"kubernetes.io/projected/01b63947-ab2c-4052-91a9-f91a1a499168-istio-token\") pod \"01b63947-ab2c-4052-91a9-f91a1a499168\" (UID: \"01b63947-ab2c-4052-91a9-f91a1a499168\") "
Oct 30 05:19:51.426445 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:51.426427    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01b63947-ab2c-4052-91a9-f91a1a499168-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01b63947-ab2c-4052-91a9-f91a1a499168-workload-socket\") pod \"01b63947-ab2c-4052-91a9-f91a1a499168\" (UID: \"01b63947-ab2c-4052-91a9-f91a1a499168\") "
Oct 30 05:19:51.426579 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:51.426563    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01b63947-ab2c-4052-91a9-f91a1a499168-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01b63947-ab2c-4052-91a9-f91a1a499168-istio-data\") pod \"01b63947-ab2c-4052-91a9-f91a1a499168\" (UID: \"01b63947-ab2c-4052-91a9-f91a1a499168\") "
Oct 30 05:19:51.426746 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:51.426729    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01b63947-ab2c-4052-91a9-f91a1a499168-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01b63947-ab2c-4052-91a9-f91a1a499168-istiod-ca-cert\") pod \"01b63947-ab2c-4052-91a9-f91a1a499168\" (UID: \"01b63947-ab2c-4052-91a9-f91a1a499168\") "
Oct 30 05:19:51.426992 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:51.426973    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01b63947-ab2c-4052-91a9-f91a1a499168-kube-api-access-nstq6\" (UniqueName: \"kubernetes.io/projected/01b63947-ab2c-4052-91a9-f91a1a499168-kube-api-access-nstq6\") pod \"01b63947-ab2c-4052-91a9-f91a1a499168\" (UID: \"01b63947-ab2c-4052-91a9-f91a1a499168\") "
Oct 30 05:19:51.427150 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:51.427134    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01b63947-ab2c-4052-91a9-f91a1a499168-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01b63947-ab2c-4052-91a9-f91a1a499168-credential-socket\") pod \"01b63947-ab2c-4052-91a9-f91a1a499168\" (UID: \"01b63947-ab2c-4052-91a9-f91a1a499168\") "
Oct 30 05:19:51.428294 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:51.428270    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01b63947-ab2c-4052-91a9-f91a1a499168-workload-socket" pod "01b63947-ab2c-4052-91a9-f91a1a499168" (UID: "01b63947-ab2c-4052-91a9-f91a1a499168"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:19:51.428400 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:51.428280    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01b63947-ab2c-4052-91a9-f91a1a499168-credential-socket" pod "01b63947-ab2c-4052-91a9-f91a1a499168" (UID: "01b63947-ab2c-4052-91a9-f91a1a499168"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:19:51.428673 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:51.428653    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01b63947-ab2c-4052-91a9-f91a1a499168-istio-data" pod "01b63947-ab2c-4052-91a9-f91a1a499168" (UID: "01b63947-ab2c-4052-91a9-f91a1a499168"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:19:51.428813 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:51.428738    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01b63947-ab2c-4052-91a9-f91a1a499168-workload-certs" pod "01b63947-ab2c-4052-91a9-f91a1a499168" (UID: "01b63947-ab2c-4052-91a9-f91a1a499168"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:19:51.429078 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:51.429053    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01b63947-ab2c-4052-91a9-f91a1a499168-istiod-ca-cert" pod "01b63947-ab2c-4052-91a9-f91a1a499168" (UID: "01b63947-ab2c-4052-91a9-f91a1a499168"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:19:51.431615 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:51.431590    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01b63947-ab2c-4052-91a9-f91a1a499168-istio-token" pod "01b63947-ab2c-4052-91a9-f91a1a499168" (UID: "01b63947-ab2c-4052-91a9-f91a1a499168"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:19:51.433229 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:51.433203    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01b63947-ab2c-4052-91a9-f91a1a499168-kube-api-access-nstq6" pod "01b63947-ab2c-4052-91a9-f91a1a499168" (UID: "01b63947-ab2c-4052-91a9-f91a1a499168"). InnerVolumeSpecName "kube-api-access-nstq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:19:51.446288 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:51.446188    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/01b63947-ab2c-4052-91a9-f91a1a499168-istio-podinfo" pod "01b63947-ab2c-4052-91a9-f91a1a499168" (UID: "01b63947-ab2c-4052-91a9-f91a1a499168"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:19:51.448180 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:51.448154    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01b63947-ab2c-4052-91a9-f91a1a499168-istio-envoy" pod "01b63947-ab2c-4052-91a9-f91a1a499168" (UID: "01b63947-ab2c-4052-91a9-f91a1a499168"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:19:51.530139 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:51.530101    5898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01b63947-ab2c-4052-91a9-f91a1a499168-istiod-ca-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:19:51.530139 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:51.530136    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nstq6\" (UniqueName: \"kubernetes.io/projected/01b63947-ab2c-4052-91a9-f91a1a499168-kube-api-access-nstq6\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:19:51.530349 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:51.530151    5898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01b63947-ab2c-4052-91a9-f91a1a499168-credential-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:19:51.530349 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:51.530165    5898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01b63947-ab2c-4052-91a9-f91a1a499168-istio-podinfo\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:19:51.530349 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:51.530179    5898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01b63947-ab2c-4052-91a9-f91a1a499168-istio-envoy\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:19:51.530349 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:51.530191    5898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01b63947-ab2c-4052-91a9-f91a1a499168-workload-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:19:51.530349 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:51.530204    5898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01b63947-ab2c-4052-91a9-f91a1a499168-istio-token\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:19:51.530349 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:51.530217    5898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01b63947-ab2c-4052-91a9-f91a1a499168-workload-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:19:51.530349 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:51.530228    5898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01b63947-ab2c-4052-91a9-f91a1a499168-istio-data\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:19:51.924763 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:51.924734    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw"
Oct 30 05:19:51.974151 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:51.974115    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:19:52.039081 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:52.039004    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8aed7491-445f-4c4a-86c1-3c844692deee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aed7491-445f-4c4a-86c1-3c844692deee-istio-podinfo\") pod \"8aed7491-445f-4c4a-86c1-3c844692deee\" (UID: \"8aed7491-445f-4c4a-86c1-3c844692deee\") "
Oct 30 05:19:52.039081 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:52.039041    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aed7491-445f-4c4a-86c1-3c844692deee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aed7491-445f-4c4a-86c1-3c844692deee-istio-data\") pod \"8aed7491-445f-4c4a-86c1-3c844692deee\" (UID: \"8aed7491-445f-4c4a-86c1-3c844692deee\") "
Oct 30 05:19:52.039081 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:52.039070    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8aed7491-445f-4c4a-86c1-3c844692deee-istio-token\" (UniqueName: \"kubernetes.io/projected/8aed7491-445f-4c4a-86c1-3c844692deee-istio-token\") pod \"8aed7491-445f-4c4a-86c1-3c844692deee\" (UID: \"8aed7491-445f-4c4a-86c1-3c844692deee\") "
Oct 30 05:19:52.039372 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:52.039128    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aed7491-445f-4c4a-86c1-3c844692deee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aed7491-445f-4c4a-86c1-3c844692deee-workload-socket\") pod \"8aed7491-445f-4c4a-86c1-3c844692deee\" (UID: \"8aed7491-445f-4c4a-86c1-3c844692deee\") "
Oct 30 05:19:52.039372 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:52.039152    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aed7491-445f-4c4a-86c1-3c844692deee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aed7491-445f-4c4a-86c1-3c844692deee-workload-certs\") pod \"8aed7491-445f-4c4a-86c1-3c844692deee\" (UID: \"8aed7491-445f-4c4a-86c1-3c844692deee\") "
Oct 30 05:19:52.039372 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:52.039216    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aed7491-445f-4c4a-86c1-3c844692deee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aed7491-445f-4c4a-86c1-3c844692deee-credential-socket\") pod \"8aed7491-445f-4c4a-86c1-3c844692deee\" (UID: \"8aed7491-445f-4c4a-86c1-3c844692deee\") "
Oct 30 05:19:52.039544 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:52.039372    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8aed7491-445f-4c4a-86c1-3c844692deee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aed7491-445f-4c4a-86c1-3c844692deee-istiod-ca-cert\") pod \"8aed7491-445f-4c4a-86c1-3c844692deee\" (UID: \"8aed7491-445f-4c4a-86c1-3c844692deee\") "
Oct 30 05:19:52.039544 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:52.039387    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aed7491-445f-4c4a-86c1-3c844692deee-istio-data" pod "8aed7491-445f-4c4a-86c1-3c844692deee" (UID: "8aed7491-445f-4c4a-86c1-3c844692deee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:19:52.039544 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:52.039484    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8aed7491-445f-4c4a-86c1-3c844692deee-kube-api-access-m4mxl\" (UniqueName: \"kubernetes.io/projected/8aed7491-445f-4c4a-86c1-3c844692deee-kube-api-access-m4mxl\") pod \"8aed7491-445f-4c4a-86c1-3c844692deee\" (UID: \"8aed7491-445f-4c4a-86c1-3c844692deee\") "
Oct 30 05:19:52.039544 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:52.039523    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aed7491-445f-4c4a-86c1-3c844692deee-credential-socket" pod "8aed7491-445f-4c4a-86c1-3c844692deee" (UID: "8aed7491-445f-4c4a-86c1-3c844692deee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:19:52.039885 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:52.039786    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aed7491-445f-4c4a-86c1-3c844692deee-istiod-ca-cert" pod "8aed7491-445f-4c4a-86c1-3c844692deee" (UID: "8aed7491-445f-4c4a-86c1-3c844692deee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:19:52.040099 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:52.040070    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aed7491-445f-4c4a-86c1-3c844692deee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aed7491-445f-4c4a-86c1-3c844692deee-istio-envoy\") pod \"8aed7491-445f-4c4a-86c1-3c844692deee\" (UID: \"8aed7491-445f-4c4a-86c1-3c844692deee\") "
Oct 30 05:19:52.040220 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:52.040100    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aed7491-445f-4c4a-86c1-3c844692deee-workload-socket" pod "8aed7491-445f-4c4a-86c1-3c844692deee" (UID: "8aed7491-445f-4c4a-86c1-3c844692deee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:19:52.040220 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:52.040211    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aed7491-445f-4c4a-86c1-3c844692deee-workload-certs" pod "8aed7491-445f-4c4a-86c1-3c844692deee" (UID: "8aed7491-445f-4c4a-86c1-3c844692deee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:19:52.040956 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:52.040932    5898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aed7491-445f-4c4a-86c1-3c844692deee-credential-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:19:52.041052 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:52.040959    5898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aed7491-445f-4c4a-86c1-3c844692deee-istiod-ca-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:19:52.041052 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:52.040976    5898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aed7491-445f-4c4a-86c1-3c844692deee-istio-data\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:19:52.041052 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:52.040987    5898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aed7491-445f-4c4a-86c1-3c844692deee-workload-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:19:52.041052 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:52.040999    5898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aed7491-445f-4c4a-86c1-3c844692deee-workload-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:19:52.041865 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:52.041839    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aed7491-445f-4c4a-86c1-3c844692deee-istio-token" pod "8aed7491-445f-4c4a-86c1-3c844692deee" (UID: "8aed7491-445f-4c4a-86c1-3c844692deee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:19:52.042790 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:52.042761    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aed7491-445f-4c4a-86c1-3c844692deee-kube-api-access-m4mxl" pod "8aed7491-445f-4c4a-86c1-3c844692deee" (UID: "8aed7491-445f-4c4a-86c1-3c844692deee"). InnerVolumeSpecName "kube-api-access-m4mxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:19:52.042871 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:52.042809    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aed7491-445f-4c4a-86c1-3c844692deee-istio-envoy" pod "8aed7491-445f-4c4a-86c1-3c844692deee" (UID: "8aed7491-445f-4c4a-86c1-3c844692deee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:19:52.048858 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:52.048820    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8aed7491-445f-4c4a-86c1-3c844692deee-istio-podinfo" pod "8aed7491-445f-4c4a-86c1-3c844692deee" (UID: "8aed7491-445f-4c4a-86c1-3c844692deee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:19:52.076575 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:52.076484    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:19:52.078614 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:52.078585    5898 generic.go:356] "Generic (PLEG): container finished" podID="01b63947-ab2c-4052-91a9-f91a1a499168" containerID="b27bd1182ec2712d85a1e46a896fc322e493c44280df6cda6df4c6a59fab14d4" exitCode=0
Oct 30 05:19:52.078728 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:52.078623    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89" event={"ID":"01b63947-ab2c-4052-91a9-f91a1a499168","Type":"ContainerDied","Data":"b27bd1182ec2712d85a1e46a896fc322e493c44280df6cda6df4c6a59fab14d4"}
Oct 30 05:19:52.078728 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:52.078670    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89"
Oct 30 05:19:52.078728 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:52.078709    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89" event={"ID":"01b63947-ab2c-4052-91a9-f91a1a499168","Type":"ContainerDied","Data":"cb6cbd44d7abbd46568a93e64deffe13fd39e886c2ce3d23df4ac3322256ffe7"}
Oct 30 05:19:52.078933 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:52.078739    5898 scope.go:122] "RemoveContainer" containerID="b27bd1182ec2712d85a1e46a896fc322e493c44280df6cda6df4c6a59fab14d4"
Oct 30 05:19:52.080570 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:52.080553    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw"
Oct 30 05:19:52.080742 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:52.080550    5898 generic.go:356] "Generic (PLEG): container finished" podID="8aed7491-445f-4c4a-86c1-3c844692deee" containerID="54f3d0f249e07023ad63a3e0495ab1f3504347e51b5c2a8e4f0067dad6268861" exitCode=0
Oct 30 05:19:52.080851 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:52.080579    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw" event={"ID":"8aed7491-445f-4c4a-86c1-3c844692deee","Type":"ContainerDied","Data":"54f3d0f249e07023ad63a3e0495ab1f3504347e51b5c2a8e4f0067dad6268861"}
Oct 30 05:19:52.080851 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:52.080791    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw" event={"ID":"8aed7491-445f-4c4a-86c1-3c844692deee","Type":"ContainerDied","Data":"ab0233fb00b1fb6a00d237b606db244cbcbcf363530d47ed2e9fd8cb88eb4c7e"}
Oct 30 05:19:52.097892 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:52.097862    5898 scope.go:122] "RemoveContainer" containerID="b27bd1182ec2712d85a1e46a896fc322e493c44280df6cda6df4c6a59fab14d4"
Oct 30 05:19:52.098286 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:19:52.098259    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b27bd1182ec2712d85a1e46a896fc322e493c44280df6cda6df4c6a59fab14d4\": container with ID starting with b27bd1182ec2712d85a1e46a896fc322e493c44280df6cda6df4c6a59fab14d4 not found: ID does not exist" containerID="b27bd1182ec2712d85a1e46a896fc322e493c44280df6cda6df4c6a59fab14d4"
Oct 30 05:19:52.098438 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:52.098405    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b27bd1182ec2712d85a1e46a896fc322e493c44280df6cda6df4c6a59fab14d4"} err="failed to get container status \"b27bd1182ec2712d85a1e46a896fc322e493c44280df6cda6df4c6a59fab14d4\": rpc error: code = NotFound desc = could not find container \"b27bd1182ec2712d85a1e46a896fc322e493c44280df6cda6df4c6a59fab14d4\": container with ID starting with b27bd1182ec2712d85a1e46a896fc322e493c44280df6cda6df4c6a59fab14d4 not found: ID does not exist"
Oct 30 05:19:52.098527 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:52.098440    5898 scope.go:122] "RemoveContainer" containerID="54f3d0f249e07023ad63a3e0495ab1f3504347e51b5c2a8e4f0067dad6268861"
Oct 30 05:19:52.109151 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:52.109133    5898 scope.go:122] "RemoveContainer" containerID="54f3d0f249e07023ad63a3e0495ab1f3504347e51b5c2a8e4f0067dad6268861"
Oct 30 05:19:52.109391 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:19:52.109370    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54f3d0f249e07023ad63a3e0495ab1f3504347e51b5c2a8e4f0067dad6268861\": container with ID starting with 54f3d0f249e07023ad63a3e0495ab1f3504347e51b5c2a8e4f0067dad6268861 not found: ID does not exist" containerID="54f3d0f249e07023ad63a3e0495ab1f3504347e51b5c2a8e4f0067dad6268861"
Oct 30 05:19:52.109448 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:52.109398    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54f3d0f249e07023ad63a3e0495ab1f3504347e51b5c2a8e4f0067dad6268861"} err="failed to get container status \"54f3d0f249e07023ad63a3e0495ab1f3504347e51b5c2a8e4f0067dad6268861\": rpc error: code = NotFound desc = could not find container \"54f3d0f249e07023ad63a3e0495ab1f3504347e51b5c2a8e4f0067dad6268861\": container with ID starting with 54f3d0f249e07023ad63a3e0495ab1f3504347e51b5c2a8e4f0067dad6268861 not found: ID does not exist"
Oct 30 05:19:52.135662 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:52.135642    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw"]
Oct 30 05:19:52.140684 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:52.140664    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw"]
Oct 30 05:19:52.142545 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:52.142519    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4mxl\" (UniqueName: \"kubernetes.io/projected/8aed7491-445f-4c4a-86c1-3c844692deee-kube-api-access-m4mxl\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:19:52.142645 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:52.142552    5898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aed7491-445f-4c4a-86c1-3c844692deee-istio-envoy\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:19:52.142645 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:52.142568    5898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aed7491-445f-4c4a-86c1-3c844692deee-istio-podinfo\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:19:52.142645 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:52.142581    5898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8aed7491-445f-4c4a-86c1-3c844692deee-istio-token\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:19:52.187043 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:52.187012    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89"]
Oct 30 05:19:52.199809 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:52.199779    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-ngz89"]
Oct 30 05:19:52.236865 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:52.236841    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:19:52.843623 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:52.843588    5898 patch_prober.go:29] interesting pod/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 05:19:52.844046 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:52.843660    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-36fd3c-openshift-default-cc6c6c87f-pjwcw" podUID="8aed7491-445f-4c4a-86c1-3c844692deee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:19:52.891747 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:52.891719    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01b63947-ab2c-4052-91a9-f91a1a499168" path="/var/lib/kubelet/pods/01b63947-ab2c-4052-91a9-f91a1a499168/volumes"
Oct 30 05:19:52.892264 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:52.892250    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8aed7491-445f-4c4a-86c1-3c844692deee" path="/var/lib/kubelet/pods/8aed7491-445f-4c4a-86c1-3c844692deee/volumes"
Oct 30 05:19:54.287295 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:54.287252    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:19:54.386612 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:54.386579    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:19:54.486555 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:54.486514    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:19:56.066084 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:56.066047    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:19:56.296005 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:56.292054    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c7f962/test-model-c7f962-kserve-5fd5595fd9-jbf8l"]
Oct 30 05:19:56.304624 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:56.304580    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:19:56.819163 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:19:56.819129    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:19:58.000774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:58.000732    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:19:59.766846 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:59.766792    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:19:59.866834 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:19:59.866800    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:20:01.715043 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:01.714464    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:20:03.405233 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:03.405180    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:20:05.200244 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:05.200206    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:20:07.135909 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:07.134984    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:20:08.028743 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:08.028689    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:20:08.038678 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:08.038645    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-453511/test-model-453511-kserve-54dcc5c589-wmt2z"]
Oct 30 05:20:09.133773 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:09.133725    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:20:11.029163 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:11.029123    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:20:11.319960 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:11.319848    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-c7f962/test-model-c7f962-kserve-5fd5595fd9-jbf8l" podUID="e9aed90e-03b0-49b0-b0c5-509fd95836d6" containerName="main" containerID="cri-o://4c2780b744eb506d5817cb75fc0902320f6b7537315eeda90c07dc5efef54fb0" gracePeriod=45
Oct 30 05:20:11.403043 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:11.403011    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-c7f962/test-model-c7f962-kserve-5fd5595fd9-jbf8l" podUID="e9aed90e-03b0-49b0-b0c5-509fd95836d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.128:8000/ready\": dial tcp 10.133.0.128:8000: connect: connection refused"
Oct 30 05:20:11.613211 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:11.613186    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c7f962/test-model-c7f962-kserve-5fd5595fd9-jbf8l"
Oct 30 05:20:11.688944 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:11.688889    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9aed90e-03b0-49b0-b0c5-509fd95836d6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9aed90e-03b0-49b0-b0c5-509fd95836d6-tmp-dir\") pod \"e9aed90e-03b0-49b0-b0c5-509fd95836d6\" (UID: \"e9aed90e-03b0-49b0-b0c5-509fd95836d6\") "
Oct 30 05:20:11.689186 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:11.689155    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9aed90e-03b0-49b0-b0c5-509fd95836d6-tmp-dir" pod "e9aed90e-03b0-49b0-b0c5-509fd95836d6" (UID: "e9aed90e-03b0-49b0-b0c5-509fd95836d6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:20:11.689316 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:11.689280    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9aed90e-03b0-49b0-b0c5-509fd95836d6-tls-certs\" (UniqueName: \"kubernetes.io/secret/e9aed90e-03b0-49b0-b0c5-509fd95836d6-tls-certs\") pod \"e9aed90e-03b0-49b0-b0c5-509fd95836d6\" (UID: \"e9aed90e-03b0-49b0-b0c5-509fd95836d6\") "
Oct 30 05:20:11.689419 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:11.689401    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9aed90e-03b0-49b0-b0c5-509fd95836d6-home\" (UniqueName: \"kubernetes.io/empty-dir/e9aed90e-03b0-49b0-b0c5-509fd95836d6-home\") pod \"e9aed90e-03b0-49b0-b0c5-509fd95836d6\" (UID: \"e9aed90e-03b0-49b0-b0c5-509fd95836d6\") "
Oct 30 05:20:11.689782 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:11.689759    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9aed90e-03b0-49b0-b0c5-509fd95836d6-home" pod "e9aed90e-03b0-49b0-b0c5-509fd95836d6" (UID: "e9aed90e-03b0-49b0-b0c5-509fd95836d6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:20:11.689924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:11.689800    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9aed90e-03b0-49b0-b0c5-509fd95836d6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9aed90e-03b0-49b0-b0c5-509fd95836d6-dshm\") pod \"e9aed90e-03b0-49b0-b0c5-509fd95836d6\" (UID: \"e9aed90e-03b0-49b0-b0c5-509fd95836d6\") "
Oct 30 05:20:11.690127 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:11.690103    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9aed90e-03b0-49b0-b0c5-509fd95836d6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9aed90e-03b0-49b0-b0c5-509fd95836d6-model-cache\") pod \"e9aed90e-03b0-49b0-b0c5-509fd95836d6\" (UID: \"e9aed90e-03b0-49b0-b0c5-509fd95836d6\") "
Oct 30 05:20:11.690239 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:11.690154    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9aed90e-03b0-49b0-b0c5-509fd95836d6-kube-api-access-nwpxx\" (UniqueName: \"kubernetes.io/projected/e9aed90e-03b0-49b0-b0c5-509fd95836d6-kube-api-access-nwpxx\") pod \"e9aed90e-03b0-49b0-b0c5-509fd95836d6\" (UID: \"e9aed90e-03b0-49b0-b0c5-509fd95836d6\") "
Oct 30 05:20:11.690858 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:11.690682    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9aed90e-03b0-49b0-b0c5-509fd95836d6-model-cache" pod "e9aed90e-03b0-49b0-b0c5-509fd95836d6" (UID: "e9aed90e-03b0-49b0-b0c5-509fd95836d6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:20:11.691013 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:11.690896    5898 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9aed90e-03b0-49b0-b0c5-509fd95836d6-home\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:20:11.691136 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:11.691115    5898 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9aed90e-03b0-49b0-b0c5-509fd95836d6-tmp-dir\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:20:11.692134 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:11.692108    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9aed90e-03b0-49b0-b0c5-509fd95836d6-tls-certs" pod "e9aed90e-03b0-49b0-b0c5-509fd95836d6" (UID: "e9aed90e-03b0-49b0-b0c5-509fd95836d6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:20:11.692432 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:11.692405    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9aed90e-03b0-49b0-b0c5-509fd95836d6-dshm" pod "e9aed90e-03b0-49b0-b0c5-509fd95836d6" (UID: "e9aed90e-03b0-49b0-b0c5-509fd95836d6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:20:11.692633 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:11.692613    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9aed90e-03b0-49b0-b0c5-509fd95836d6-kube-api-access-nwpxx" pod "e9aed90e-03b0-49b0-b0c5-509fd95836d6" (UID: "e9aed90e-03b0-49b0-b0c5-509fd95836d6"). InnerVolumeSpecName "kube-api-access-nwpxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:20:11.792616 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:11.792582    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwpxx\" (UniqueName: \"kubernetes.io/projected/e9aed90e-03b0-49b0-b0c5-509fd95836d6-kube-api-access-nwpxx\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:20:11.792616 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:11.792612    5898 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9aed90e-03b0-49b0-b0c5-509fd95836d6-tls-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:20:11.792813 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:11.792626    5898 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9aed90e-03b0-49b0-b0c5-509fd95836d6-dshm\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:20:11.792813 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:11.792639    5898 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9aed90e-03b0-49b0-b0c5-509fd95836d6-model-cache\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:20:12.319741 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:12.319705    5898 generic.go:356] "Generic (PLEG): container finished" podID="e9aed90e-03b0-49b0-b0c5-509fd95836d6" containerID="4c2780b744eb506d5817cb75fc0902320f6b7537315eeda90c07dc5efef54fb0" exitCode=0
Oct 30 05:20:12.320252 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:12.319762    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c7f962/test-model-c7f962-kserve-5fd5595fd9-jbf8l"
Oct 30 05:20:12.320252 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:12.319784    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c7f962/test-model-c7f962-kserve-5fd5595fd9-jbf8l" event={"ID":"e9aed90e-03b0-49b0-b0c5-509fd95836d6","Type":"ContainerDied","Data":"4c2780b744eb506d5817cb75fc0902320f6b7537315eeda90c07dc5efef54fb0"}
Oct 30 05:20:12.320252 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:12.319828    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c7f962/test-model-c7f962-kserve-5fd5595fd9-jbf8l" event={"ID":"e9aed90e-03b0-49b0-b0c5-509fd95836d6","Type":"ContainerDied","Data":"21efe5157a54b0079e21b8cbdb971de03f11da3afe49bb67aa50ff51eb751912"}
Oct 30 05:20:12.320252 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:12.319845    5898 scope.go:122] "RemoveContainer" containerID="4c2780b744eb506d5817cb75fc0902320f6b7537315eeda90c07dc5efef54fb0"
Oct 30 05:20:12.333058 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:12.333035    5898 scope.go:122] "RemoveContainer" containerID="4c2780b744eb506d5817cb75fc0902320f6b7537315eeda90c07dc5efef54fb0"
Oct 30 05:20:12.333375 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:20:12.333353    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c2780b744eb506d5817cb75fc0902320f6b7537315eeda90c07dc5efef54fb0\": container with ID starting with 4c2780b744eb506d5817cb75fc0902320f6b7537315eeda90c07dc5efef54fb0 not found: ID does not exist" containerID="4c2780b744eb506d5817cb75fc0902320f6b7537315eeda90c07dc5efef54fb0"
Oct 30 05:20:12.333464 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:12.333381    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c2780b744eb506d5817cb75fc0902320f6b7537315eeda90c07dc5efef54fb0"} err="failed to get container status \"4c2780b744eb506d5817cb75fc0902320f6b7537315eeda90c07dc5efef54fb0\": rpc error: code = NotFound desc = could not find container \"4c2780b744eb506d5817cb75fc0902320f6b7537315eeda90c07dc5efef54fb0\": container with ID starting with 4c2780b744eb506d5817cb75fc0902320f6b7537315eeda90c07dc5efef54fb0 not found: ID does not exist"
Oct 30 05:20:12.372965 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:12.372940    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c7f962/test-model-c7f962-kserve-5fd5595fd9-jbf8l"]
Oct 30 05:20:12.380295 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:12.380274    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-c7f962/test-model-c7f962-kserve-5fd5595fd9-jbf8l"]
Oct 30 05:20:12.699777 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:12.699743    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:20:12.941926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:12.941229    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9aed90e-03b0-49b0-b0c5-509fd95836d6" path="/var/lib/kubelet/pods/e9aed90e-03b0-49b0-b0c5-509fd95836d6/volumes"
Oct 30 05:20:13.282930 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:13.282198    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:20:13.408371 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:13.408337    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:20:13.601039 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:13.601010    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:20:13.946057 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:13.945974    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:20:15.655763 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:15.655727    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:20:17.443551 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:17.443518    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:20:18.774537 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:18.774501    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:20:18.874755 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:18.874715    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:20:19.270706 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:19.270671    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:20:20.164192 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:20.164156    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:20:20.279778 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:20.279749    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:20:20.597854 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:20.595140    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:20:20.603134 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:20.603097    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-mx96q"]
Oct 30 05:20:20.618571 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:20.618322    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-mx96q" podUID="bb90d6b7-a751-48d5-8d58-98c021a69145" containerName="istio-proxy" containerID="cri-o://c0c1099f6596b8fbbd6e3c33e9ecd44a017e3fe8c148977b397668e84a7ff086" gracePeriod=30
Oct 30 05:20:21.576319 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:21.576276    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:20:23.109926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:23.106858    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-453511/test-model-453511-kserve-54dcc5c589-wmt2z" podUID="0e1a48a6-bdd2-46ba-8ddb-15ebca14289a" containerName="main" containerID="cri-o://bcb648d88b38cde0b2d7e5fb22aa9b1844b6ecd64bca48ca4cd61903428568c3" gracePeriod=45
Oct 30 05:20:23.146960 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:23.146892    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-453511/test-model-453511-kserve-54dcc5c589-wmt2z" podUID="0e1a48a6-bdd2-46ba-8ddb-15ebca14289a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.139:8000/ready\": dial tcp 10.133.0.139:8000: connect: connection refused"
Oct 30 05:20:23.549730 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:23.549692    5898 generic.go:356] "Generic (PLEG): container finished" podID="0e1a48a6-bdd2-46ba-8ddb-15ebca14289a" containerID="bcb648d88b38cde0b2d7e5fb22aa9b1844b6ecd64bca48ca4cd61903428568c3" exitCode=0
Oct 30 05:20:23.558950 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:23.550112    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-453511/test-model-453511-kserve-54dcc5c589-wmt2z" event={"ID":"0e1a48a6-bdd2-46ba-8ddb-15ebca14289a","Type":"ContainerDied","Data":"bcb648d88b38cde0b2d7e5fb22aa9b1844b6ecd64bca48ca4cd61903428568c3"}
Oct 30 05:20:23.559472 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:23.559434    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-453511/test-model-453511-kserve-54dcc5c589-wmt2z" event={"ID":"0e1a48a6-bdd2-46ba-8ddb-15ebca14289a","Type":"ContainerDied","Data":"642647c66343f028e15ad6e35dd2900708ad72d459972957b1183067482cc3a1"}
Oct 30 05:20:23.559666 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:23.559650    5898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="642647c66343f028e15ad6e35dd2900708ad72d459972957b1183067482cc3a1"
Oct 30 05:20:23.606251 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:23.606143    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-453511/test-model-453511-kserve-54dcc5c589-wmt2z"
Oct 30 05:20:23.693893 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:23.691684    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:20:23.815204 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:23.809563    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e1a48a6-bdd2-46ba-8ddb-15ebca14289a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e1a48a6-bdd2-46ba-8ddb-15ebca14289a-dshm\") pod \"0e1a48a6-bdd2-46ba-8ddb-15ebca14289a\" (UID: \"0e1a48a6-bdd2-46ba-8ddb-15ebca14289a\") "
Oct 30 05:20:23.815400 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:23.815289    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e1a48a6-bdd2-46ba-8ddb-15ebca14289a-kube-api-access-5kzcv\" (UniqueName: \"kubernetes.io/projected/0e1a48a6-bdd2-46ba-8ddb-15ebca14289a-kube-api-access-5kzcv\") pod \"0e1a48a6-bdd2-46ba-8ddb-15ebca14289a\" (UID: \"0e1a48a6-bdd2-46ba-8ddb-15ebca14289a\") "
Oct 30 05:20:23.815400 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:23.815358    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e1a48a6-bdd2-46ba-8ddb-15ebca14289a-tls-certs\" (UniqueName: \"kubernetes.io/secret/0e1a48a6-bdd2-46ba-8ddb-15ebca14289a-tls-certs\") pod \"0e1a48a6-bdd2-46ba-8ddb-15ebca14289a\" (UID: \"0e1a48a6-bdd2-46ba-8ddb-15ebca14289a\") "
Oct 30 05:20:23.815542 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:23.815406    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e1a48a6-bdd2-46ba-8ddb-15ebca14289a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e1a48a6-bdd2-46ba-8ddb-15ebca14289a-model-cache\") pod \"0e1a48a6-bdd2-46ba-8ddb-15ebca14289a\" (UID: \"0e1a48a6-bdd2-46ba-8ddb-15ebca14289a\") "
Oct 30 05:20:23.815613 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:23.815554    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e1a48a6-bdd2-46ba-8ddb-15ebca14289a-home\" (UniqueName: \"kubernetes.io/empty-dir/0e1a48a6-bdd2-46ba-8ddb-15ebca14289a-home\") pod \"0e1a48a6-bdd2-46ba-8ddb-15ebca14289a\" (UID: \"0e1a48a6-bdd2-46ba-8ddb-15ebca14289a\") "
Oct 30 05:20:23.815678 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:23.815655    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e1a48a6-bdd2-46ba-8ddb-15ebca14289a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e1a48a6-bdd2-46ba-8ddb-15ebca14289a-tmp-dir\") pod \"0e1a48a6-bdd2-46ba-8ddb-15ebca14289a\" (UID: \"0e1a48a6-bdd2-46ba-8ddb-15ebca14289a\") "
Oct 30 05:20:23.822999 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:23.822345    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e1a48a6-bdd2-46ba-8ddb-15ebca14289a-dshm" pod "0e1a48a6-bdd2-46ba-8ddb-15ebca14289a" (UID: "0e1a48a6-bdd2-46ba-8ddb-15ebca14289a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:20:23.822999 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:23.822366    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e1a48a6-bdd2-46ba-8ddb-15ebca14289a-model-cache" pod "0e1a48a6-bdd2-46ba-8ddb-15ebca14289a" (UID: "0e1a48a6-bdd2-46ba-8ddb-15ebca14289a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:20:23.822999 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:23.822570    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e1a48a6-bdd2-46ba-8ddb-15ebca14289a-home" pod "0e1a48a6-bdd2-46ba-8ddb-15ebca14289a" (UID: "0e1a48a6-bdd2-46ba-8ddb-15ebca14289a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:20:23.822999 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:23.822885    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e1a48a6-bdd2-46ba-8ddb-15ebca14289a-tmp-dir" pod "0e1a48a6-bdd2-46ba-8ddb-15ebca14289a" (UID: "0e1a48a6-bdd2-46ba-8ddb-15ebca14289a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:20:23.824349 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:23.824325    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e1a48a6-bdd2-46ba-8ddb-15ebca14289a-tls-certs" pod "0e1a48a6-bdd2-46ba-8ddb-15ebca14289a" (UID: "0e1a48a6-bdd2-46ba-8ddb-15ebca14289a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:20:23.827884 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:23.827853    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e1a48a6-bdd2-46ba-8ddb-15ebca14289a-kube-api-access-5kzcv" pod "0e1a48a6-bdd2-46ba-8ddb-15ebca14289a" (UID: "0e1a48a6-bdd2-46ba-8ddb-15ebca14289a"). InnerVolumeSpecName "kube-api-access-5kzcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:20:23.838087 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:23.834416    5898 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e1a48a6-bdd2-46ba-8ddb-15ebca14289a-home\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:20:23.838087 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:23.834488    5898 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e1a48a6-bdd2-46ba-8ddb-15ebca14289a-tmp-dir\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:20:23.838087 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:23.834503    5898 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e1a48a6-bdd2-46ba-8ddb-15ebca14289a-dshm\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:20:23.838087 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:23.834519    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kzcv\" (UniqueName: \"kubernetes.io/projected/0e1a48a6-bdd2-46ba-8ddb-15ebca14289a-kube-api-access-5kzcv\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:20:23.838087 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:23.834541    5898 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e1a48a6-bdd2-46ba-8ddb-15ebca14289a-tls-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:20:23.838087 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:23.834556    5898 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e1a48a6-bdd2-46ba-8ddb-15ebca14289a-model-cache\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:20:24.557303 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:24.557226    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-453511/test-model-453511-kserve-54dcc5c589-wmt2z"
Oct 30 05:20:24.608672 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:24.608636    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-453511/test-model-453511-kserve-54dcc5c589-wmt2z"]
Oct 30 05:20:24.619418 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:24.619389    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-453511/test-model-453511-kserve-54dcc5c589-wmt2z"]
Oct 30 05:20:24.892013 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:24.891982    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e1a48a6-bdd2-46ba-8ddb-15ebca14289a" path="/var/lib/kubelet/pods/0e1a48a6-bdd2-46ba-8ddb-15ebca14289a/volumes"
Oct 30 05:20:25.006523 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:25.006478    5898 patch_prober.go:29] interesting pod/e2e-inf-a-010013-openshift-default-59977f9669-mx96q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 05:20:25.006688 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:25.006543    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-mx96q" podUID="bb90d6b7-a751-48d5-8d58-98c021a69145" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 05:20:25.039843 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:25.039804    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:20:25.180794 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:25.180703    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:20:25.484285 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:25.484195    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:20:26.050767 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:26.050734    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-mx96q"
Oct 30 05:20:26.173425 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:26.173394    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb90d6b7-a751-48d5-8d58-98c021a69145-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb90d6b7-a751-48d5-8d58-98c021a69145-credential-socket\") pod \"bb90d6b7-a751-48d5-8d58-98c021a69145\" (UID: \"bb90d6b7-a751-48d5-8d58-98c021a69145\") "
Oct 30 05:20:26.173637 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:26.173480    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb90d6b7-a751-48d5-8d58-98c021a69145-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb90d6b7-a751-48d5-8d58-98c021a69145-istiod-ca-cert\") pod \"bb90d6b7-a751-48d5-8d58-98c021a69145\" (UID: \"bb90d6b7-a751-48d5-8d58-98c021a69145\") "
Oct 30 05:20:26.173637 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:26.173572    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb90d6b7-a751-48d5-8d58-98c021a69145-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb90d6b7-a751-48d5-8d58-98c021a69145-istio-podinfo\") pod \"bb90d6b7-a751-48d5-8d58-98c021a69145\" (UID: \"bb90d6b7-a751-48d5-8d58-98c021a69145\") "
Oct 30 05:20:26.173637 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:26.173609    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb90d6b7-a751-48d5-8d58-98c021a69145-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb90d6b7-a751-48d5-8d58-98c021a69145-istio-data\") pod \"bb90d6b7-a751-48d5-8d58-98c021a69145\" (UID: \"bb90d6b7-a751-48d5-8d58-98c021a69145\") "
Oct 30 05:20:26.173806 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:26.173661    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb90d6b7-a751-48d5-8d58-98c021a69145-kube-api-access-8dk64\" (UniqueName: \"kubernetes.io/projected/bb90d6b7-a751-48d5-8d58-98c021a69145-kube-api-access-8dk64\") pod \"bb90d6b7-a751-48d5-8d58-98c021a69145\" (UID: \"bb90d6b7-a751-48d5-8d58-98c021a69145\") "
Oct 30 05:20:26.173806 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:26.173714    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb90d6b7-a751-48d5-8d58-98c021a69145-credential-socket" pod "bb90d6b7-a751-48d5-8d58-98c021a69145" (UID: "bb90d6b7-a751-48d5-8d58-98c021a69145"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:20:26.173806 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:26.173788    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb90d6b7-a751-48d5-8d58-98c021a69145-istio-token\" (UniqueName: \"kubernetes.io/projected/bb90d6b7-a751-48d5-8d58-98c021a69145-istio-token\") pod \"bb90d6b7-a751-48d5-8d58-98c021a69145\" (UID: \"bb90d6b7-a751-48d5-8d58-98c021a69145\") "
Oct 30 05:20:26.173989 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:26.173943    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb90d6b7-a751-48d5-8d58-98c021a69145-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb90d6b7-a751-48d5-8d58-98c021a69145-istio-envoy\") pod \"bb90d6b7-a751-48d5-8d58-98c021a69145\" (UID: \"bb90d6b7-a751-48d5-8d58-98c021a69145\") "
Oct 30 05:20:26.173989 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:26.173968    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb90d6b7-a751-48d5-8d58-98c021a69145-istiod-ca-cert" pod "bb90d6b7-a751-48d5-8d58-98c021a69145" (UID: "bb90d6b7-a751-48d5-8d58-98c021a69145"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:20:26.173989 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:26.173984    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb90d6b7-a751-48d5-8d58-98c021a69145-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb90d6b7-a751-48d5-8d58-98c021a69145-workload-socket\") pod \"bb90d6b7-a751-48d5-8d58-98c021a69145\" (UID: \"bb90d6b7-a751-48d5-8d58-98c021a69145\") "
Oct 30 05:20:26.174168 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:26.173990    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb90d6b7-a751-48d5-8d58-98c021a69145-istio-data" pod "bb90d6b7-a751-48d5-8d58-98c021a69145" (UID: "bb90d6b7-a751-48d5-8d58-98c021a69145"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:20:26.174168 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:26.174035    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb90d6b7-a751-48d5-8d58-98c021a69145-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb90d6b7-a751-48d5-8d58-98c021a69145-workload-certs\") pod \"bb90d6b7-a751-48d5-8d58-98c021a69145\" (UID: \"bb90d6b7-a751-48d5-8d58-98c021a69145\") "
Oct 30 05:20:26.174268 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:26.174244    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb90d6b7-a751-48d5-8d58-98c021a69145-workload-socket" pod "bb90d6b7-a751-48d5-8d58-98c021a69145" (UID: "bb90d6b7-a751-48d5-8d58-98c021a69145"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:20:26.174356 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:26.174334    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb90d6b7-a751-48d5-8d58-98c021a69145-workload-certs" pod "bb90d6b7-a751-48d5-8d58-98c021a69145" (UID: "bb90d6b7-a751-48d5-8d58-98c021a69145"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:20:26.174768 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:26.174742    5898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb90d6b7-a751-48d5-8d58-98c021a69145-istiod-ca-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:20:26.174871 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:26.174773    5898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb90d6b7-a751-48d5-8d58-98c021a69145-istio-data\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:20:26.174871 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:26.174786    5898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb90d6b7-a751-48d5-8d58-98c021a69145-workload-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:20:26.174871 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:26.174798    5898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb90d6b7-a751-48d5-8d58-98c021a69145-workload-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:20:26.174871 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:26.174812    5898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb90d6b7-a751-48d5-8d58-98c021a69145-credential-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:20:26.176053 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:26.176032    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb90d6b7-a751-48d5-8d58-98c021a69145-istio-podinfo" pod "bb90d6b7-a751-48d5-8d58-98c021a69145" (UID: "bb90d6b7-a751-48d5-8d58-98c021a69145"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:20:26.176241 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:26.176225    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb90d6b7-a751-48d5-8d58-98c021a69145-kube-api-access-8dk64" pod "bb90d6b7-a751-48d5-8d58-98c021a69145" (UID: "bb90d6b7-a751-48d5-8d58-98c021a69145"). InnerVolumeSpecName "kube-api-access-8dk64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:20:26.176344 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:26.176266    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb90d6b7-a751-48d5-8d58-98c021a69145-istio-envoy" pod "bb90d6b7-a751-48d5-8d58-98c021a69145" (UID: "bb90d6b7-a751-48d5-8d58-98c021a69145"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:20:26.176523 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:26.176499    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb90d6b7-a751-48d5-8d58-98c021a69145-istio-token" pod "bb90d6b7-a751-48d5-8d58-98c021a69145" (UID: "bb90d6b7-a751-48d5-8d58-98c021a69145"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:20:26.276611 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:26.276535    5898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb90d6b7-a751-48d5-8d58-98c021a69145-istio-envoy\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:20:26.276611 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:26.276562    5898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb90d6b7-a751-48d5-8d58-98c021a69145-istio-podinfo\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:20:26.276611 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:26.276574    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dk64\" (UniqueName: \"kubernetes.io/projected/bb90d6b7-a751-48d5-8d58-98c021a69145-kube-api-access-8dk64\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:20:26.276611 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:26.276583    5898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb90d6b7-a751-48d5-8d58-98c021a69145-istio-token\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:20:26.577445 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:26.577411    5898 generic.go:356] "Generic (PLEG): container finished" podID="bb90d6b7-a751-48d5-8d58-98c021a69145" containerID="c0c1099f6596b8fbbd6e3c33e9ecd44a017e3fe8c148977b397668e84a7ff086" exitCode=0
Oct 30 05:20:26.577627 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:26.577457    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-mx96q"
Oct 30 05:20:26.577627 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:26.577473    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-mx96q" event={"ID":"bb90d6b7-a751-48d5-8d58-98c021a69145","Type":"ContainerDied","Data":"c0c1099f6596b8fbbd6e3c33e9ecd44a017e3fe8c148977b397668e84a7ff086"}
Oct 30 05:20:26.577627 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:26.577527    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-mx96q" event={"ID":"bb90d6b7-a751-48d5-8d58-98c021a69145","Type":"ContainerDied","Data":"fdc475c8108045f5a5303665bfad0eaddbf6bb2d0be4ce09fb20e0d5e8adb09c"}
Oct 30 05:20:26.577627 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:26.577558    5898 scope.go:122] "RemoveContainer" containerID="c0c1099f6596b8fbbd6e3c33e9ecd44a017e3fe8c148977b397668e84a7ff086"
Oct 30 05:20:26.589939 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:26.589917    5898 scope.go:122] "RemoveContainer" containerID="c0c1099f6596b8fbbd6e3c33e9ecd44a017e3fe8c148977b397668e84a7ff086"
Oct 30 05:20:26.590222 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:20:26.590204    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0c1099f6596b8fbbd6e3c33e9ecd44a017e3fe8c148977b397668e84a7ff086\": container with ID starting with c0c1099f6596b8fbbd6e3c33e9ecd44a017e3fe8c148977b397668e84a7ff086 not found: ID does not exist" containerID="c0c1099f6596b8fbbd6e3c33e9ecd44a017e3fe8c148977b397668e84a7ff086"
Oct 30 05:20:26.590293 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:26.590232    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0c1099f6596b8fbbd6e3c33e9ecd44a017e3fe8c148977b397668e84a7ff086"} err="failed to get container status \"c0c1099f6596b8fbbd6e3c33e9ecd44a017e3fe8c148977b397668e84a7ff086\": rpc error: code = NotFound desc = could not find container \"c0c1099f6596b8fbbd6e3c33e9ecd44a017e3fe8c148977b397668e84a7ff086\": container with ID starting with c0c1099f6596b8fbbd6e3c33e9ecd44a017e3fe8c148977b397668e84a7ff086 not found: ID does not exist"
Oct 30 05:20:26.640943 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:26.640915    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-mx96q"]
Oct 30 05:20:26.652228 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:26.652197    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-010013-openshift-default-59977f9669-mx96q"]
Oct 30 05:20:26.818077 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:20:26.818043    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:20:26.893195 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:26.893108    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb90d6b7-a751-48d5-8d58-98c021a69145" path="/var/lib/kubelet/pods/bb90d6b7-a751-48d5-8d58-98c021a69145/volumes"
Oct 30 05:20:27.077372 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:27.077339    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:20:27.237573 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:27.237481    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:20:29.017995 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:29.017964    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:20:30.966793 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:30.966756    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:20:31.680470 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:31.680438    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:20:31.810374 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:31.810332    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:20:32.146003 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:32.145964    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:20:32.871997 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:32.871917    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:20:34.774756 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:34.774719    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:20:36.520448 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:36.520398    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:20:38.154167 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:38.154137    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:20:38.278639 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:38.278602    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:20:39.827242 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:39.827206    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34722e-554d6dcb9f-rxxd9"]
Oct 30 05:20:39.827974 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:39.827953    5898 cpu_manager.go:414] "containerMap: removing container" podUID="01b63947-ab2c-4052-91a9-f91a1a499168" containerName="istio-proxy"
Oct 30 05:20:39.827974 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:39.827974    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01b63947-ab2c-4052-91a9-f91a1a499168" containerName="istio-proxy"
Oct 30 05:20:39.828106 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:39.827994    5898 cpu_manager.go:414] "containerMap: removing container" podUID="0e1a48a6-bdd2-46ba-8ddb-15ebca14289a" containerName="main"
Oct 30 05:20:39.828106 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:39.828000    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e1a48a6-bdd2-46ba-8ddb-15ebca14289a" containerName="main"
Oct 30 05:20:39.828106 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:39.828019    5898 cpu_manager.go:414] "containerMap: removing container" podUID="e9aed90e-03b0-49b0-b0c5-509fd95836d6" containerName="main"
Oct 30 05:20:39.828106 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:39.828023    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9aed90e-03b0-49b0-b0c5-509fd95836d6" containerName="main"
Oct 30 05:20:39.828106 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:39.828042    5898 cpu_manager.go:414] "containerMap: removing container" podUID="983a192c-67a4-4684-86fc-6df70f38a1ed" containerName="istio-proxy"
Oct 30 05:20:39.828106 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:39.828047    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="983a192c-67a4-4684-86fc-6df70f38a1ed" containerName="istio-proxy"
Oct 30 05:20:39.828106 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:39.828060    5898 cpu_manager.go:414] "containerMap: removing container" podUID="bb90d6b7-a751-48d5-8d58-98c021a69145" containerName="istio-proxy"
Oct 30 05:20:39.828106 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:39.828064    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb90d6b7-a751-48d5-8d58-98c021a69145" containerName="istio-proxy"
Oct 30 05:20:39.828106 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:39.828076    5898 cpu_manager.go:414] "containerMap: removing container" podUID="b5d063ac-6da8-4217-bddd-82a05db7bc50" containerName="istio-proxy"
Oct 30 05:20:39.828106 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:39.828081    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5d063ac-6da8-4217-bddd-82a05db7bc50" containerName="istio-proxy"
Oct 30 05:20:39.828106 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:39.828090    5898 cpu_manager.go:414] "containerMap: removing container" podUID="8aed7491-445f-4c4a-86c1-3c844692deee" containerName="istio-proxy"
Oct 30 05:20:39.828106 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:39.828094    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8aed7491-445f-4c4a-86c1-3c844692deee" containerName="istio-proxy"
Oct 30 05:20:39.828106 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:39.828101    5898 cpu_manager.go:414] "containerMap: removing container" podUID="fac1e366-ef3a-45e5-bd96-8ebf290f0b5f" containerName="main"
Oct 30 05:20:39.828106 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:39.828105    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fac1e366-ef3a-45e5-bd96-8ebf290f0b5f" containerName="main"
Oct 30 05:20:39.828774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:39.828118    5898 cpu_manager.go:414] "containerMap: removing container" podUID="ef140ba8-eb01-40bd-9b95-216c8bacd7af" containerName="main"
Oct 30 05:20:39.828774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:39.828122    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef140ba8-eb01-40bd-9b95-216c8bacd7af" containerName="main"
Oct 30 05:20:39.828774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:39.828207    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb90d6b7-a751-48d5-8d58-98c021a69145" containerName="istio-proxy"
Oct 30 05:20:39.828774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:39.828216    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5d063ac-6da8-4217-bddd-82a05db7bc50" containerName="istio-proxy"
Oct 30 05:20:39.828774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:39.828223    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="8aed7491-445f-4c4a-86c1-3c844692deee" containerName="istio-proxy"
Oct 30 05:20:39.828774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:39.828228    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="fac1e366-ef3a-45e5-bd96-8ebf290f0b5f" containerName="main"
Oct 30 05:20:39.828774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:39.828235    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef140ba8-eb01-40bd-9b95-216c8bacd7af" containerName="main"
Oct 30 05:20:39.828774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:39.828240    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="01b63947-ab2c-4052-91a9-f91a1a499168" containerName="istio-proxy"
Oct 30 05:20:39.828774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:39.828249    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e1a48a6-bdd2-46ba-8ddb-15ebca14289a" containerName="main"
Oct 30 05:20:39.828774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:39.828259    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9aed90e-03b0-49b0-b0c5-509fd95836d6" containerName="main"
Oct 30 05:20:39.828774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:39.828268    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="983a192c-67a4-4684-86fc-6df70f38a1ed" containerName="istio-proxy"
Oct 30 05:20:39.833620 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:39.833597    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34722e-554d6dcb9f-rxxd9"
Oct 30 05:20:39.851277 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:39.851250    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34722e-554d6dcb9f-rxxd9"]
Oct 30 05:20:39.853076 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:39.853049    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-34722e-554d6dcb9f-rxxd9\" (UID: \"8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34722e-554d6dcb9f-rxxd9"
Oct 30 05:20:39.853222 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:39.853094    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-34722e-554d6dcb9f-rxxd9\" (UID: \"8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34722e-554d6dcb9f-rxxd9"
Oct 30 05:20:39.853410 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:39.853385    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-34722e-554d6dcb9f-rxxd9\" (UID: \"8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34722e-554d6dcb9f-rxxd9"
Oct 30 05:20:39.853563 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:39.853539    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-994nm\" (UniqueName: \"kubernetes.io/projected/8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff-kube-api-access-994nm\") pod \"maas-api-e2e-shared-a-w5-34722e-554d6dcb9f-rxxd9\" (UID: \"8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34722e-554d6dcb9f-rxxd9"
Oct 30 05:20:39.863186 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:39.863159    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w5-34722e-serving-cert\""
Oct 30 05:20:39.909671 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:39.909638    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:20:39.955048 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:39.955022    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-994nm\" (UniqueName: \"kubernetes.io/projected/8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff-kube-api-access-994nm\") pod \"maas-api-e2e-shared-a-w5-34722e-554d6dcb9f-rxxd9\" (UID: \"8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34722e-554d6dcb9f-rxxd9"
Oct 30 05:20:39.955226 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:39.955066    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-34722e-554d6dcb9f-rxxd9\" (UID: \"8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34722e-554d6dcb9f-rxxd9"
Oct 30 05:20:39.955226 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:39.955094    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-34722e-554d6dcb9f-rxxd9\" (UID: \"8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34722e-554d6dcb9f-rxxd9"
Oct 30 05:20:39.955441 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:39.955419    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-34722e-554d6dcb9f-rxxd9\" (UID: \"8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34722e-554d6dcb9f-rxxd9"
Oct 30 05:20:39.956167 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:39.956141    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-34722e-554d6dcb9f-rxxd9\" (UID: \"8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34722e-554d6dcb9f-rxxd9"
Oct 30 05:20:39.957652 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:39.957627    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-34722e-554d6dcb9f-rxxd9\" (UID: \"8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34722e-554d6dcb9f-rxxd9"
Oct 30 05:20:39.957836 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:39.957813    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-34722e-554d6dcb9f-rxxd9\" (UID: \"8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34722e-554d6dcb9f-rxxd9"
Oct 30 05:20:39.975605 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:39.975572    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-994nm\" (UniqueName: \"kubernetes.io/projected/8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff-kube-api-access-994nm\") pod \"maas-api-e2e-shared-a-w5-34722e-554d6dcb9f-rxxd9\" (UID: \"8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34722e-554d6dcb9f-rxxd9"
Oct 30 05:20:39.998249 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:39.998223    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:20:40.100481 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:40.100382    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:20:40.150766 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:40.150735    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34722e-554d6dcb9f-rxxd9"
Oct 30 05:20:40.460036 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:40.460013    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34722e-554d6dcb9f-rxxd9"]
Oct 30 05:20:40.462337 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:20:40.462311    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d64c096_d3e4_4ef4_bc1a_08d5dfc97fff.slice/crio-25440a5e63171687771c23c10e2a1989b1d264b217d8b343b6a8235eddfa04db WatchSource:0}: Error finding container 25440a5e63171687771c23c10e2a1989b1d264b217d8b343b6a8235eddfa04db: Status 404 returned error can't find the container with id 25440a5e63171687771c23c10e2a1989b1d264b217d8b343b6a8235eddfa04db
Oct 30 05:20:40.463838 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:40.463818    5898 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 05:20:40.682968 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:40.682928    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34722e-554d6dcb9f-rxxd9" event={"ID":"8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff","Type":"ContainerStarted","Data":"25440a5e63171687771c23c10e2a1989b1d264b217d8b343b6a8235eddfa04db"}
Oct 30 05:20:41.691214 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:41.691167    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34722e-554d6dcb9f-rxxd9" event={"ID":"8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff","Type":"ContainerStarted","Data":"5a11dc9dfdf7726122edca5ca0898744893254a179be1830a4bef948d731b0a4"}
Oct 30 05:20:41.691712 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:41.691257    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34722e-554d6dcb9f-rxxd9"
Oct 30 05:20:41.729660 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:41.729596    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34722e-554d6dcb9f-rxxd9" podStartSLOduration=2.396401613 podStartE2EDuration="2.729578952s" totalImagesPullingTime="333.177339ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:20:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:20:41.727450262 +0000 UTC m=+2170.951270196" watchObservedRunningTime="2026-10-30 05:20:41.729578952 +0000 UTC m=+2170.953398856"
Oct 30 05:20:41.755656 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:41.755625    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:20:41.882370 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:41.882315    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:20:43.693847 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:43.693812    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:20:45.651745 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:45.651710    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:20:47.386355 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:47.386326    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:20:47.703855 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:47.703777    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34722e-554d6dcb9f-rxxd9"
Oct 30 05:20:49.129569 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:49.129534    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:20:50.869813 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:50.869783    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:20:52.562277 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:52.562246    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:20:54.241490 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:54.241459    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:20:55.715179 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:55.715147    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:20:56.372909 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:56.372867    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:20:56.494243 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:56.494209    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:20:56.599025 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:56.598989    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:20:56.819406 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:20:56.819320    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:20:57.058082 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:57.058028    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:20:58.481299 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:58.481264    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:20:58.778355 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:20:58.778258    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:00.311941 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:00.311896    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:01.974186 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:01.974150    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:03.141752 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:03.141716    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:03.235723 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:03.235692    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:03.582453 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:03.582413    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:03.730272 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:03.730048    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:05.560140 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:05.560105    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:07.123840 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:07.123808    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8afb26-688849cfdf-6b6c5"]
Oct 30 05:21:07.124454 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:07.124079    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8afb26-688849cfdf-6b6c5" podUID="96952584-fdf3-4731-a845-83ec28cd69b4" containerName="maas-api" containerID="cri-o://184c6e2e5ffb159849fc19bb80e2f772872e90300a27f4117924d3d34d1941c3" gracePeriod=30
Oct 30 05:21:07.177764 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:07.177732    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:07.578870 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:07.572396    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8afb26-688849cfdf-6b6c5"
Oct 30 05:21:07.712086 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:07.710560    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96952584-fdf3-4731-a845-83ec28cd69b4-kube-api-access-qpp82\" (UniqueName: \"kubernetes.io/projected/96952584-fdf3-4731-a845-83ec28cd69b4-kube-api-access-qpp82\") pod \"96952584-fdf3-4731-a845-83ec28cd69b4\" (UID: \"96952584-fdf3-4731-a845-83ec28cd69b4\") "
Oct 30 05:21:07.712086 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:07.710625    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96952584-fdf3-4731-a845-83ec28cd69b4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96952584-fdf3-4731-a845-83ec28cd69b4-openshift-service-ca\") pod \"96952584-fdf3-4731-a845-83ec28cd69b4\" (UID: \"96952584-fdf3-4731-a845-83ec28cd69b4\") "
Oct 30 05:21:07.712086 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:07.710921    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96952584-fdf3-4731-a845-83ec28cd69b4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96952584-fdf3-4731-a845-83ec28cd69b4-maas-api-tls\") pod \"96952584-fdf3-4731-a845-83ec28cd69b4\" (UID: \"96952584-fdf3-4731-a845-83ec28cd69b4\") "
Oct 30 05:21:07.712086 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:07.711016    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96952584-fdf3-4731-a845-83ec28cd69b4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/96952584-fdf3-4731-a845-83ec28cd69b4-metrics-certs\") pod \"96952584-fdf3-4731-a845-83ec28cd69b4\" (UID: \"96952584-fdf3-4731-a845-83ec28cd69b4\") "
Oct 30 05:21:07.715083 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:07.714892    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96952584-fdf3-4731-a845-83ec28cd69b4-metrics-certs" pod "96952584-fdf3-4731-a845-83ec28cd69b4" (UID: "96952584-fdf3-4731-a845-83ec28cd69b4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:21:07.715267 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:07.715209    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96952584-fdf3-4731-a845-83ec28cd69b4-openshift-service-ca" pod "96952584-fdf3-4731-a845-83ec28cd69b4" (UID: "96952584-fdf3-4731-a845-83ec28cd69b4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:21:07.719088 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:07.719047    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96952584-fdf3-4731-a845-83ec28cd69b4-maas-api-tls" pod "96952584-fdf3-4731-a845-83ec28cd69b4" (UID: "96952584-fdf3-4731-a845-83ec28cd69b4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:21:07.719683 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:07.719647    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96952584-fdf3-4731-a845-83ec28cd69b4-kube-api-access-qpp82" pod "96952584-fdf3-4731-a845-83ec28cd69b4" (UID: "96952584-fdf3-4731-a845-83ec28cd69b4"). InnerVolumeSpecName "kube-api-access-qpp82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:21:07.790855 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:07.790773    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:07.813073 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:07.813048    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96952584-fdf3-4731-a845-83ec28cd69b4-maas-api-tls\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:21:07.813073 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:07.813072    5898 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96952584-fdf3-4731-a845-83ec28cd69b4-metrics-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:21:07.813240 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:07.813081    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpp82\" (UniqueName: \"kubernetes.io/projected/96952584-fdf3-4731-a845-83ec28cd69b4-kube-api-access-qpp82\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:21:07.813240 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:07.813091    5898 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96952584-fdf3-4731-a845-83ec28cd69b4-openshift-service-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:21:07.906671 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:07.906638    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:07.993080 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:07.993048    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:08.009183 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:08.009149    5898 generic.go:356] "Generic (PLEG): container finished" podID="96952584-fdf3-4731-a845-83ec28cd69b4" containerID="184c6e2e5ffb159849fc19bb80e2f772872e90300a27f4117924d3d34d1941c3" exitCode=0
Oct 30 05:21:08.009334 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:08.009198    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8afb26-688849cfdf-6b6c5"
Oct 30 05:21:08.009334 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:08.009231    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8afb26-688849cfdf-6b6c5" event={"ID":"96952584-fdf3-4731-a845-83ec28cd69b4","Type":"ContainerDied","Data":"184c6e2e5ffb159849fc19bb80e2f772872e90300a27f4117924d3d34d1941c3"}
Oct 30 05:21:08.009334 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:08.009285    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8afb26-688849cfdf-6b6c5" event={"ID":"96952584-fdf3-4731-a845-83ec28cd69b4","Type":"ContainerDied","Data":"f889aa3073e97f7404cf8ffa328de86fb5dd9f3fd441631300228c0930b36c91"}
Oct 30 05:21:08.009334 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:08.009307    5898 scope.go:122] "RemoveContainer" containerID="184c6e2e5ffb159849fc19bb80e2f772872e90300a27f4117924d3d34d1941c3"
Oct 30 05:21:08.020939 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:08.020920    5898 scope.go:122] "RemoveContainer" containerID="184c6e2e5ffb159849fc19bb80e2f772872e90300a27f4117924d3d34d1941c3"
Oct 30 05:21:08.021220 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:21:08.021200    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"184c6e2e5ffb159849fc19bb80e2f772872e90300a27f4117924d3d34d1941c3\": container with ID starting with 184c6e2e5ffb159849fc19bb80e2f772872e90300a27f4117924d3d34d1941c3 not found: ID does not exist" containerID="184c6e2e5ffb159849fc19bb80e2f772872e90300a27f4117924d3d34d1941c3"
Oct 30 05:21:08.021273 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:08.021229    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"184c6e2e5ffb159849fc19bb80e2f772872e90300a27f4117924d3d34d1941c3"} err="failed to get container status \"184c6e2e5ffb159849fc19bb80e2f772872e90300a27f4117924d3d34d1941c3\": rpc error: code = NotFound desc = could not find container \"184c6e2e5ffb159849fc19bb80e2f772872e90300a27f4117924d3d34d1941c3\": container with ID starting with 184c6e2e5ffb159849fc19bb80e2f772872e90300a27f4117924d3d34d1941c3 not found: ID does not exist"
Oct 30 05:21:08.068311 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:08.068213    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8afb26-688849cfdf-6b6c5"]
Oct 30 05:21:08.077927 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:08.077867    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8afb26-688849cfdf-6b6c5"]
Oct 30 05:21:08.493462 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:08.493431    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:08.892274 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:08.892238    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96952584-fdf3-4731-a845-83ec28cd69b4" path="/var/lib/kubelet/pods/96952584-fdf3-4731-a845-83ec28cd69b4/volumes"
Oct 30 05:21:09.827117 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:09.827085    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:11.749645 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:11.749604    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:13.454545 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:13.454513    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:14.432930 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:14.432833    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:14.533738 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:14.533697    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:14.716299 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:14.716204    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:14.813599 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:14.813566    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:14.896232 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:14.896190    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:15.373363 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:15.373331    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:21:15.411878 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:15.411781    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:15.748396 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:15.748305    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:16.269960 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:16.269928    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:16.505222 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:16.505189    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:17.600824 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:17.600789    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:19.485215 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:19.485182    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:19.896482 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:19.896452    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:19.999743 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:19.999711    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:21.774532 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:21.774494    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:21.875084 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:21.875050    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:23.884337 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:23.884307    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:25.581783 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:25.581750    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:26.818425 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:21:26.818389    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:21:27.378607 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:27.378573    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:28.757027 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:28.756978    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:29.475138 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:29.475099    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:29.643653 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:29.643610    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:29.951696 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:29.951661    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:30.425781 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:30.425753    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:30.619924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:30.618840    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:31.671191 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:31.671149    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:32.657658 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:32.657626    5898 scope.go:122] "RemoveContainer" containerID="bcb648d88b38cde0b2d7e5fb22aa9b1844b6ecd64bca48ca4cd61903428568c3"
Oct 30 05:21:33.468774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:33.468742    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:34.137137 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:34.137101    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:34.244263 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:34.244228    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:35.403767 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:35.403735    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:35.938689 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:35.938658    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:36.193153 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:36.193067    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:37.328237 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:37.328205    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:38.975883 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:38.975842    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:39.608662 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:39.608627    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:39.708713 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:39.708678    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:40.855630 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:40.855591    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:41.869747 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:41.869711    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:42.536330 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:42.536297    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:44.029836 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:44.029799    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:44.648045 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:44.647997    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:44.767350 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:44.767273    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:44.887238 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:44.887196    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:45.331534 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:45.331499    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:47.038644 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:47.038614    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:48.471655 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:48.471616    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:49.194223 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:49.194191    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:49.345718 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:49.345683    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:49.451812 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:49.451738    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:49.865793 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:49.865760    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:51.260120 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:51.260087    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:51.492490 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:51.492462    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:51.617841 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:51.617808    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:51.939757 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:51.939674    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:53.080577 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:53.080543    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:54.932603 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:54.932566    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:55.503674 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:55.503631    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-356cae/test-model-356cae-kserve-78bbbb868f-bmnn8"]
Oct 30 05:21:55.534424 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:55.532310    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:56.202559 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:56.202489    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:56.339314 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:56.339282    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:56.537668 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:56.537588    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:56.723731 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:56.723653    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:21:56.847455 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:21:56.847417    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:21:58.658529 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:21:58.658497    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:00.502751 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:00.502697    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:00.740250 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:00.740188    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:02.310107 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:02.310067    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:02.410637 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:02.410606    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:02.510644 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:02.510611    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:04.104676 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:04.104646    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:05.854693 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:05.854640    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:07.599936 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:07.599867    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:07.693620 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:07.693590    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:09.314729 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:09.314695    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:10.544713 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:10.544665    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-356cae/test-model-356cae-kserve-78bbbb868f-bmnn8" podUID="ee5dfade-6b22-4653-80d9-75320b4c6117" containerName="main" containerID="cri-o://f98c5af65814cbcfd8b764f127433a568ac3688d64af0090424eb077e4ca293e" gracePeriod=45
Oct 30 05:22:10.670688 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:10.670651    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-356cae/test-model-356cae-kserve-78bbbb868f-bmnn8" podUID="ee5dfade-6b22-4653-80d9-75320b4c6117" containerName="main" probeResult="failure" output="Get \"https://10.133.0.147:8000/ready\": dial tcp 10.133.0.147:8000: connect: connection refused"
Oct 30 05:22:10.830461 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:10.830424    5898 generic.go:356] "Generic (PLEG): container finished" podID="ee5dfade-6b22-4653-80d9-75320b4c6117" containerID="f98c5af65814cbcfd8b764f127433a568ac3688d64af0090424eb077e4ca293e" exitCode=0
Oct 30 05:22:10.830602 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:10.830491    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-356cae/test-model-356cae-kserve-78bbbb868f-bmnn8" event={"ID":"ee5dfade-6b22-4653-80d9-75320b4c6117","Type":"ContainerDied","Data":"f98c5af65814cbcfd8b764f127433a568ac3688d64af0090424eb077e4ca293e"}
Oct 30 05:22:10.830602 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:10.830532    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-356cae/test-model-356cae-kserve-78bbbb868f-bmnn8" event={"ID":"ee5dfade-6b22-4653-80d9-75320b4c6117","Type":"ContainerDied","Data":"8961abc881ee04d4ab5804ebc1ec16e1abb15f68b41859406859bc9c07fa6ab8"}
Oct 30 05:22:10.830602 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:10.830544    5898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8961abc881ee04d4ab5804ebc1ec16e1abb15f68b41859406859bc9c07fa6ab8"
Oct 30 05:22:10.837370 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:10.837349    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-356cae/test-model-356cae-kserve-78bbbb868f-bmnn8"
Oct 30 05:22:10.926326 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:10.926277    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee5dfade-6b22-4653-80d9-75320b4c6117-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee5dfade-6b22-4653-80d9-75320b4c6117-model-cache\") pod \"ee5dfade-6b22-4653-80d9-75320b4c6117\" (UID: \"ee5dfade-6b22-4653-80d9-75320b4c6117\") "
Oct 30 05:22:10.926513 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:10.926348    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee5dfade-6b22-4653-80d9-75320b4c6117-home\" (UniqueName: \"kubernetes.io/empty-dir/ee5dfade-6b22-4653-80d9-75320b4c6117-home\") pod \"ee5dfade-6b22-4653-80d9-75320b4c6117\" (UID: \"ee5dfade-6b22-4653-80d9-75320b4c6117\") "
Oct 30 05:22:10.926513 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:10.926494    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee5dfade-6b22-4653-80d9-75320b4c6117-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee5dfade-6b22-4653-80d9-75320b4c6117-dshm\") pod \"ee5dfade-6b22-4653-80d9-75320b4c6117\" (UID: \"ee5dfade-6b22-4653-80d9-75320b4c6117\") "
Oct 30 05:22:10.926708 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:10.926541    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee5dfade-6b22-4653-80d9-75320b4c6117-tls-certs\" (UniqueName: \"kubernetes.io/secret/ee5dfade-6b22-4653-80d9-75320b4c6117-tls-certs\") pod \"ee5dfade-6b22-4653-80d9-75320b4c6117\" (UID: \"ee5dfade-6b22-4653-80d9-75320b4c6117\") "
Oct 30 05:22:10.926708 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:10.926662    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee5dfade-6b22-4653-80d9-75320b4c6117-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee5dfade-6b22-4653-80d9-75320b4c6117-tmp-dir\") pod \"ee5dfade-6b22-4653-80d9-75320b4c6117\" (UID: \"ee5dfade-6b22-4653-80d9-75320b4c6117\") "
Oct 30 05:22:10.926819 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:10.926714    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee5dfade-6b22-4653-80d9-75320b4c6117-kube-api-access-68zvk\" (UniqueName: \"kubernetes.io/projected/ee5dfade-6b22-4653-80d9-75320b4c6117-kube-api-access-68zvk\") pod \"ee5dfade-6b22-4653-80d9-75320b4c6117\" (UID: \"ee5dfade-6b22-4653-80d9-75320b4c6117\") "
Oct 30 05:22:10.927230 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:10.927180    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee5dfade-6b22-4653-80d9-75320b4c6117-model-cache" pod "ee5dfade-6b22-4653-80d9-75320b4c6117" (UID: "ee5dfade-6b22-4653-80d9-75320b4c6117"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:22:10.927376 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:10.927330    5898 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee5dfade-6b22-4653-80d9-75320b4c6117-model-cache\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:22:10.927547 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:10.927528    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee5dfade-6b22-4653-80d9-75320b4c6117-home" pod "ee5dfade-6b22-4653-80d9-75320b4c6117" (UID: "ee5dfade-6b22-4653-80d9-75320b4c6117"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:22:10.927700 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:10.927681    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee5dfade-6b22-4653-80d9-75320b4c6117-tmp-dir" pod "ee5dfade-6b22-4653-80d9-75320b4c6117" (UID: "ee5dfade-6b22-4653-80d9-75320b4c6117"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:22:10.929533 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:10.929507    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee5dfade-6b22-4653-80d9-75320b4c6117-tls-certs" pod "ee5dfade-6b22-4653-80d9-75320b4c6117" (UID: "ee5dfade-6b22-4653-80d9-75320b4c6117"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:22:10.929889 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:10.929860    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee5dfade-6b22-4653-80d9-75320b4c6117-kube-api-access-68zvk" pod "ee5dfade-6b22-4653-80d9-75320b4c6117" (UID: "ee5dfade-6b22-4653-80d9-75320b4c6117"). InnerVolumeSpecName "kube-api-access-68zvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:22:10.929889 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:10.929873    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee5dfade-6b22-4653-80d9-75320b4c6117-dshm" pod "ee5dfade-6b22-4653-80d9-75320b4c6117" (UID: "ee5dfade-6b22-4653-80d9-75320b4c6117"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:22:11.028237 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:11.028200    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68zvk\" (UniqueName: \"kubernetes.io/projected/ee5dfade-6b22-4653-80d9-75320b4c6117-kube-api-access-68zvk\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:22:11.028237 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:11.028229    5898 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee5dfade-6b22-4653-80d9-75320b4c6117-home\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:22:11.028237 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:11.028241    5898 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee5dfade-6b22-4653-80d9-75320b4c6117-dshm\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:22:11.028473 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:11.028252    5898 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee5dfade-6b22-4653-80d9-75320b4c6117-tls-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:22:11.028473 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:11.028265    5898 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee5dfade-6b22-4653-80d9-75320b4c6117-tmp-dir\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:22:11.144219 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:11.144185    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:11.837509 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:11.837476    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-356cae/test-model-356cae-kserve-78bbbb868f-bmnn8"
Oct 30 05:22:11.869232 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:11.869200    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-e16297/test-model-e16297-kserve-5654446c8c-lwfxr"]
Oct 30 05:22:11.893170 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:11.893133    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-356cae/test-model-356cae-kserve-78bbbb868f-bmnn8"]
Oct 30 05:22:11.895424 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:11.895394    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:11.899633 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:11.899607    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-356cae/test-model-356cae-kserve-78bbbb868f-bmnn8"]
Oct 30 05:22:12.892544 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:12.892511    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee5dfade-6b22-4653-80d9-75320b4c6117" path="/var/lib/kubelet/pods/ee5dfade-6b22-4653-80d9-75320b4c6117/volumes"
Oct 30 05:22:13.075625 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:13.075589    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:14.541246 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:14.541212    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:22:15.165357 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:15.165323    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:17.080721 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:17.080684    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:17.178377 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:17.178349    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:19.066109 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:19.066074    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:20.761315 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:20.761282    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:22.510625 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:22.510591    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:22.691924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:22.674498    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:22.775533 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:22.775446    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:24.208475 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:24.208424    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:25.862190 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:25.862154    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:26.819789 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:22:26.819753    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:22:26.895487 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:26.895454    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-e16297/test-model-e16297-kserve-5654446c8c-lwfxr" podUID="4d4f8e87-1d0e-434a-9be6-d66558d209eb" containerName="main" containerID="cri-o://da4011e3554a4183ea205f92f3fd34ff8e6e258c47feee5227a06f95e844ea1d" gracePeriod=45
Oct 30 05:22:27.194496 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:27.194466    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e16297/test-model-e16297-kserve-5654446c8c-lwfxr"
Oct 30 05:22:27.272998 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:27.272963    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d4f8e87-1d0e-434a-9be6-d66558d209eb-tls-certs\" (UniqueName: \"kubernetes.io/secret/4d4f8e87-1d0e-434a-9be6-d66558d209eb-tls-certs\") pod \"4d4f8e87-1d0e-434a-9be6-d66558d209eb\" (UID: \"4d4f8e87-1d0e-434a-9be6-d66558d209eb\") "
Oct 30 05:22:27.273192 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:27.273031    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d4f8e87-1d0e-434a-9be6-d66558d209eb-kube-api-access-5pwh8\" (UniqueName: \"kubernetes.io/projected/4d4f8e87-1d0e-434a-9be6-d66558d209eb-kube-api-access-5pwh8\") pod \"4d4f8e87-1d0e-434a-9be6-d66558d209eb\" (UID: \"4d4f8e87-1d0e-434a-9be6-d66558d209eb\") "
Oct 30 05:22:27.273192 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:27.273075    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d4f8e87-1d0e-434a-9be6-d66558d209eb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d4f8e87-1d0e-434a-9be6-d66558d209eb-tmp-dir\") pod \"4d4f8e87-1d0e-434a-9be6-d66558d209eb\" (UID: \"4d4f8e87-1d0e-434a-9be6-d66558d209eb\") "
Oct 30 05:22:27.273192 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:27.273110    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d4f8e87-1d0e-434a-9be6-d66558d209eb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d4f8e87-1d0e-434a-9be6-d66558d209eb-model-cache\") pod \"4d4f8e87-1d0e-434a-9be6-d66558d209eb\" (UID: \"4d4f8e87-1d0e-434a-9be6-d66558d209eb\") "
Oct 30 05:22:27.273192 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:27.273187    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d4f8e87-1d0e-434a-9be6-d66558d209eb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d4f8e87-1d0e-434a-9be6-d66558d209eb-dshm\") pod \"4d4f8e87-1d0e-434a-9be6-d66558d209eb\" (UID: \"4d4f8e87-1d0e-434a-9be6-d66558d209eb\") "
Oct 30 05:22:27.273412 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:27.273212    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d4f8e87-1d0e-434a-9be6-d66558d209eb-home\" (UniqueName: \"kubernetes.io/empty-dir/4d4f8e87-1d0e-434a-9be6-d66558d209eb-home\") pod \"4d4f8e87-1d0e-434a-9be6-d66558d209eb\" (UID: \"4d4f8e87-1d0e-434a-9be6-d66558d209eb\") "
Oct 30 05:22:27.273624 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:27.273598    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d4f8e87-1d0e-434a-9be6-d66558d209eb-model-cache" pod "4d4f8e87-1d0e-434a-9be6-d66558d209eb" (UID: "4d4f8e87-1d0e-434a-9be6-d66558d209eb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:22:27.273841 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:27.273811    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d4f8e87-1d0e-434a-9be6-d66558d209eb-tmp-dir" pod "4d4f8e87-1d0e-434a-9be6-d66558d209eb" (UID: "4d4f8e87-1d0e-434a-9be6-d66558d209eb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:22:27.274126 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:27.274103    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d4f8e87-1d0e-434a-9be6-d66558d209eb-home" pod "4d4f8e87-1d0e-434a-9be6-d66558d209eb" (UID: "4d4f8e87-1d0e-434a-9be6-d66558d209eb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:22:27.274363 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:27.274343    5898 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d4f8e87-1d0e-434a-9be6-d66558d209eb-tmp-dir\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:22:27.274433 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:27.274369    5898 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d4f8e87-1d0e-434a-9be6-d66558d209eb-model-cache\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:22:27.274433 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:27.274383    5898 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d4f8e87-1d0e-434a-9be6-d66558d209eb-home\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:22:27.277207 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:27.277182    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d4f8e87-1d0e-434a-9be6-d66558d209eb-tls-certs" pod "4d4f8e87-1d0e-434a-9be6-d66558d209eb" (UID: "4d4f8e87-1d0e-434a-9be6-d66558d209eb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:22:27.277848 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:27.277823    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d4f8e87-1d0e-434a-9be6-d66558d209eb-kube-api-access-5pwh8" pod "4d4f8e87-1d0e-434a-9be6-d66558d209eb" (UID: "4d4f8e87-1d0e-434a-9be6-d66558d209eb"). InnerVolumeSpecName "kube-api-access-5pwh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:22:27.279013 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:27.278988    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d4f8e87-1d0e-434a-9be6-d66558d209eb-dshm" pod "4d4f8e87-1d0e-434a-9be6-d66558d209eb" (UID: "4d4f8e87-1d0e-434a-9be6-d66558d209eb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:22:27.375497 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:27.375426    5898 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d4f8e87-1d0e-434a-9be6-d66558d209eb-tls-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:22:27.375497 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:27.375452    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pwh8\" (UniqueName: \"kubernetes.io/projected/4d4f8e87-1d0e-434a-9be6-d66558d209eb-kube-api-access-5pwh8\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:22:27.375497 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:27.375463    5898 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d4f8e87-1d0e-434a-9be6-d66558d209eb-dshm\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:22:27.572876 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:27.572845    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:27.980818 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:27.980781    5898 generic.go:356] "Generic (PLEG): container finished" podID="4d4f8e87-1d0e-434a-9be6-d66558d209eb" containerID="da4011e3554a4183ea205f92f3fd34ff8e6e258c47feee5227a06f95e844ea1d" exitCode=0
Oct 30 05:22:27.981309 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:27.980831    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e16297/test-model-e16297-kserve-5654446c8c-lwfxr"
Oct 30 05:22:27.981309 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:27.980853    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e16297/test-model-e16297-kserve-5654446c8c-lwfxr" event={"ID":"4d4f8e87-1d0e-434a-9be6-d66558d209eb","Type":"ContainerDied","Data":"da4011e3554a4183ea205f92f3fd34ff8e6e258c47feee5227a06f95e844ea1d"}
Oct 30 05:22:27.981309 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:27.980890    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e16297/test-model-e16297-kserve-5654446c8c-lwfxr" event={"ID":"4d4f8e87-1d0e-434a-9be6-d66558d209eb","Type":"ContainerDied","Data":"add869577c7a4990b5cfabbcf0677472a78203d53f5c8b97bfa6cf1635bbcf3c"}
Oct 30 05:22:27.981309 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:27.980926    5898 scope.go:122] "RemoveContainer" containerID="da4011e3554a4183ea205f92f3fd34ff8e6e258c47feee5227a06f95e844ea1d"
Oct 30 05:22:27.992086 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:27.992065    5898 scope.go:122] "RemoveContainer" containerID="da4011e3554a4183ea205f92f3fd34ff8e6e258c47feee5227a06f95e844ea1d"
Oct 30 05:22:27.992360 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:22:27.992335    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da4011e3554a4183ea205f92f3fd34ff8e6e258c47feee5227a06f95e844ea1d\": container with ID starting with da4011e3554a4183ea205f92f3fd34ff8e6e258c47feee5227a06f95e844ea1d not found: ID does not exist" containerID="da4011e3554a4183ea205f92f3fd34ff8e6e258c47feee5227a06f95e844ea1d"
Oct 30 05:22:27.992466 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:27.992365    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da4011e3554a4183ea205f92f3fd34ff8e6e258c47feee5227a06f95e844ea1d"} err="failed to get container status \"da4011e3554a4183ea205f92f3fd34ff8e6e258c47feee5227a06f95e844ea1d\": rpc error: code = NotFound desc = could not find container \"da4011e3554a4183ea205f92f3fd34ff8e6e258c47feee5227a06f95e844ea1d\": container with ID starting with da4011e3554a4183ea205f92f3fd34ff8e6e258c47feee5227a06f95e844ea1d not found: ID does not exist"
Oct 30 05:22:28.030493 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:28.030442    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-e16297/test-model-e16297-kserve-5654446c8c-lwfxr"]
Oct 30 05:22:28.040953 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:28.040931    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-e16297/test-model-e16297-kserve-5654446c8c-lwfxr"]
Oct 30 05:22:28.896468 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:28.896434    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d4f8e87-1d0e-434a-9be6-d66558d209eb" path="/var/lib/kubelet/pods/4d4f8e87-1d0e-434a-9be6-d66558d209eb/volumes"
Oct 30 05:22:29.299247 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:29.299150    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:31.043947 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:31.043896    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:32.874734 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:32.874691    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:34.493228 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:34.493163    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:35.912944 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:35.912871    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:36.475472 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:36.475435    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:36.594159 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:36.594108    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:36.686084 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:36.686047    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:37.181406 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:37.181335    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:38.520677 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:38.520633    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:40.360266 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:40.360234    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:41.892934 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:41.892885    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:41.996269 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:41.996232    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:42.122931 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:42.119390    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:43.285815 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:43.285783    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:43.389187 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:43.389151    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:43.646319 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:43.646276    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p"]
Oct 30 05:22:43.646603 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:43.646576    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p" podUID="e1d94e3b-c651-4989-afbe-53acbd131f71" containerName="istio-proxy" containerID="cri-o://30a82c2d3d8a70b02954e5140ecd779b5f9f043f8aba447e23006e3d57d7f3c3" gracePeriod=30
Oct 30 05:22:43.677467 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:43.676969    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:43.869927 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:43.869503    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:44.284619 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:44.284590    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:46.023453 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:46.023423    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:46.939438 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:46.939401    5898 patch_prober.go:29] interesting pod/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 05:22:46.939613 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:46.939458    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p" podUID="e1d94e3b-c651-4989-afbe-53acbd131f71" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 05:22:47.044548 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:47.044506    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:47.537626 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:47.537587    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:47.721123 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:47.721085    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:48.031423 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:48.031384    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:48.142737 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:48.142682    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:48.249129 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:48.249098    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:48.346430 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:48.346352    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:48.457929 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:48.453775    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:48.560950 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:48.547394    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:48.798486 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:48.798387    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:48.972034 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:48.972004    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p"
Oct 30 05:22:49.007169 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:49.007135    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:49.015447 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:49.015417    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1d94e3b-c651-4989-afbe-53acbd131f71-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1d94e3b-c651-4989-afbe-53acbd131f71-credential-socket\") pod \"e1d94e3b-c651-4989-afbe-53acbd131f71\" (UID: \"e1d94e3b-c651-4989-afbe-53acbd131f71\") "
Oct 30 05:22:49.015724 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:49.015694    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1d94e3b-c651-4989-afbe-53acbd131f71-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1d94e3b-c651-4989-afbe-53acbd131f71-workload-socket\") pod \"e1d94e3b-c651-4989-afbe-53acbd131f71\" (UID: \"e1d94e3b-c651-4989-afbe-53acbd131f71\") "
Oct 30 05:22:49.015938 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:49.015920    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1d94e3b-c651-4989-afbe-53acbd131f71-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1d94e3b-c651-4989-afbe-53acbd131f71-istio-data\") pod \"e1d94e3b-c651-4989-afbe-53acbd131f71\" (UID: \"e1d94e3b-c651-4989-afbe-53acbd131f71\") "
Oct 30 05:22:49.016208 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:49.016190    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1d94e3b-c651-4989-afbe-53acbd131f71-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1d94e3b-c651-4989-afbe-53acbd131f71-istio-envoy\") pod \"e1d94e3b-c651-4989-afbe-53acbd131f71\" (UID: \"e1d94e3b-c651-4989-afbe-53acbd131f71\") "
Oct 30 05:22:49.016484 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:49.016455    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1d94e3b-c651-4989-afbe-53acbd131f71-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1d94e3b-c651-4989-afbe-53acbd131f71-istiod-ca-cert\") pod \"e1d94e3b-c651-4989-afbe-53acbd131f71\" (UID: \"e1d94e3b-c651-4989-afbe-53acbd131f71\") "
Oct 30 05:22:49.016695 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:49.016672    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1d94e3b-c651-4989-afbe-53acbd131f71-kube-api-access-pbpxd\" (UniqueName: \"kubernetes.io/projected/e1d94e3b-c651-4989-afbe-53acbd131f71-kube-api-access-pbpxd\") pod \"e1d94e3b-c651-4989-afbe-53acbd131f71\" (UID: \"e1d94e3b-c651-4989-afbe-53acbd131f71\") "
Oct 30 05:22:49.016824 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:49.016708    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1d94e3b-c651-4989-afbe-53acbd131f71-istio-token\" (UniqueName: \"kubernetes.io/projected/e1d94e3b-c651-4989-afbe-53acbd131f71-istio-token\") pod \"e1d94e3b-c651-4989-afbe-53acbd131f71\" (UID: \"e1d94e3b-c651-4989-afbe-53acbd131f71\") "
Oct 30 05:22:49.016824 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:49.016797    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1d94e3b-c651-4989-afbe-53acbd131f71-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1d94e3b-c651-4989-afbe-53acbd131f71-workload-certs\") pod \"e1d94e3b-c651-4989-afbe-53acbd131f71\" (UID: \"e1d94e3b-c651-4989-afbe-53acbd131f71\") "
Oct 30 05:22:49.017028 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:49.016847    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e1d94e3b-c651-4989-afbe-53acbd131f71-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1d94e3b-c651-4989-afbe-53acbd131f71-istio-podinfo\") pod \"e1d94e3b-c651-4989-afbe-53acbd131f71\" (UID: \"e1d94e3b-c651-4989-afbe-53acbd131f71\") "
Oct 30 05:22:49.019005 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:49.018975    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1d94e3b-c651-4989-afbe-53acbd131f71-workload-socket" pod "e1d94e3b-c651-4989-afbe-53acbd131f71" (UID: "e1d94e3b-c651-4989-afbe-53acbd131f71"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:22:49.019115 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:49.019006    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1d94e3b-c651-4989-afbe-53acbd131f71-credential-socket" pod "e1d94e3b-c651-4989-afbe-53acbd131f71" (UID: "e1d94e3b-c651-4989-afbe-53acbd131f71"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:22:49.019695 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:49.019656    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1d94e3b-c651-4989-afbe-53acbd131f71-istio-data" pod "e1d94e3b-c651-4989-afbe-53acbd131f71" (UID: "e1d94e3b-c651-4989-afbe-53acbd131f71"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:22:49.020026 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:49.019992    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1d94e3b-c651-4989-afbe-53acbd131f71-workload-certs" pod "e1d94e3b-c651-4989-afbe-53acbd131f71" (UID: "e1d94e3b-c651-4989-afbe-53acbd131f71"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:22:49.020141 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:49.020050    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1d94e3b-c651-4989-afbe-53acbd131f71-istiod-ca-cert" pod "e1d94e3b-c651-4989-afbe-53acbd131f71" (UID: "e1d94e3b-c651-4989-afbe-53acbd131f71"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:22:49.021762 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:49.021740    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1d94e3b-c651-4989-afbe-53acbd131f71-istio-token" pod "e1d94e3b-c651-4989-afbe-53acbd131f71" (UID: "e1d94e3b-c651-4989-afbe-53acbd131f71"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:22:49.022051 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:49.022023    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1d94e3b-c651-4989-afbe-53acbd131f71-istio-envoy" pod "e1d94e3b-c651-4989-afbe-53acbd131f71" (UID: "e1d94e3b-c651-4989-afbe-53acbd131f71"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:22:49.024305 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:49.024271    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e1d94e3b-c651-4989-afbe-53acbd131f71-istio-podinfo" pod "e1d94e3b-c651-4989-afbe-53acbd131f71" (UID: "e1d94e3b-c651-4989-afbe-53acbd131f71"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:22:49.026573 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:49.026539    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1d94e3b-c651-4989-afbe-53acbd131f71-kube-api-access-pbpxd" pod "e1d94e3b-c651-4989-afbe-53acbd131f71" (UID: "e1d94e3b-c651-4989-afbe-53acbd131f71"). InnerVolumeSpecName "kube-api-access-pbpxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:22:49.119480 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:49.119444    5898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1d94e3b-c651-4989-afbe-53acbd131f71-istio-data\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:22:49.119480 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:49.119480    5898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1d94e3b-c651-4989-afbe-53acbd131f71-istio-envoy\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:22:49.119694 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:49.119496    5898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1d94e3b-c651-4989-afbe-53acbd131f71-istiod-ca-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:22:49.119694 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:49.119510    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbpxd\" (UniqueName: \"kubernetes.io/projected/e1d94e3b-c651-4989-afbe-53acbd131f71-kube-api-access-pbpxd\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:22:49.119694 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:49.119524    5898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1d94e3b-c651-4989-afbe-53acbd131f71-istio-token\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:22:49.119694 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:49.119537    5898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1d94e3b-c651-4989-afbe-53acbd131f71-workload-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:22:49.119694 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:49.119549    5898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1d94e3b-c651-4989-afbe-53acbd131f71-istio-podinfo\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:22:49.119694 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:49.119562    5898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1d94e3b-c651-4989-afbe-53acbd131f71-credential-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:22:49.119694 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:49.119587    5898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1d94e3b-c651-4989-afbe-53acbd131f71-workload-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:22:49.143769 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:49.143738    5898 generic.go:356] "Generic (PLEG): container finished" podID="e1d94e3b-c651-4989-afbe-53acbd131f71" containerID="30a82c2d3d8a70b02954e5140ecd779b5f9f043f8aba447e23006e3d57d7f3c3" exitCode=0
Oct 30 05:22:49.144243 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:49.143783    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p" event={"ID":"e1d94e3b-c651-4989-afbe-53acbd131f71","Type":"ContainerDied","Data":"30a82c2d3d8a70b02954e5140ecd779b5f9f043f8aba447e23006e3d57d7f3c3"}
Oct 30 05:22:49.144243 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:49.143833    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p" event={"ID":"e1d94e3b-c651-4989-afbe-53acbd131f71","Type":"ContainerDied","Data":"613529a61e138694536993792943b1e98c6a6410993dd56960db9d88c1532a11"}
Oct 30 05:22:49.144243 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:49.143866    5898 scope.go:122] "RemoveContainer" containerID="30a82c2d3d8a70b02954e5140ecd779b5f9f043f8aba447e23006e3d57d7f3c3"
Oct 30 05:22:49.144243 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:49.143787    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p"
Oct 30 05:22:49.156304 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:49.156278    5898 scope.go:122] "RemoveContainer" containerID="d56328b64cfc56d88c83488b4f04061781785f631a042f33f74b30d94e1dae95"
Oct 30 05:22:49.168228 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:49.168211    5898 scope.go:122] "RemoveContainer" containerID="30a82c2d3d8a70b02954e5140ecd779b5f9f043f8aba447e23006e3d57d7f3c3"
Oct 30 05:22:49.168523 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:22:49.168506    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30a82c2d3d8a70b02954e5140ecd779b5f9f043f8aba447e23006e3d57d7f3c3\": container with ID starting with 30a82c2d3d8a70b02954e5140ecd779b5f9f043f8aba447e23006e3d57d7f3c3 not found: ID does not exist" containerID="30a82c2d3d8a70b02954e5140ecd779b5f9f043f8aba447e23006e3d57d7f3c3"
Oct 30 05:22:49.168585 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:49.168533    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30a82c2d3d8a70b02954e5140ecd779b5f9f043f8aba447e23006e3d57d7f3c3"} err="failed to get container status \"30a82c2d3d8a70b02954e5140ecd779b5f9f043f8aba447e23006e3d57d7f3c3\": rpc error: code = NotFound desc = could not find container \"30a82c2d3d8a70b02954e5140ecd779b5f9f043f8aba447e23006e3d57d7f3c3\": container with ID starting with 30a82c2d3d8a70b02954e5140ecd779b5f9f043f8aba447e23006e3d57d7f3c3 not found: ID does not exist"
Oct 30 05:22:49.168585 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:49.168553    5898 scope.go:122] "RemoveContainer" containerID="d56328b64cfc56d88c83488b4f04061781785f631a042f33f74b30d94e1dae95"
Oct 30 05:22:49.168835 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:22:49.168816    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d56328b64cfc56d88c83488b4f04061781785f631a042f33f74b30d94e1dae95\": container with ID starting with d56328b64cfc56d88c83488b4f04061781785f631a042f33f74b30d94e1dae95 not found: ID does not exist" containerID="d56328b64cfc56d88c83488b4f04061781785f631a042f33f74b30d94e1dae95"
Oct 30 05:22:49.168922 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:49.168838    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d56328b64cfc56d88c83488b4f04061781785f631a042f33f74b30d94e1dae95"} err="failed to get container status \"d56328b64cfc56d88c83488b4f04061781785f631a042f33f74b30d94e1dae95\": rpc error: code = NotFound desc = could not find container \"d56328b64cfc56d88c83488b4f04061781785f631a042f33f74b30d94e1dae95\": container with ID starting with d56328b64cfc56d88c83488b4f04061781785f631a042f33f74b30d94e1dae95 not found: ID does not exist"
Oct 30 05:22:49.198994 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:49.198929    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p"]
Oct 30 05:22:49.206168 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:49.206147    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-2a8f18-openshift-default-74685fbccc-g8j2p"]
Oct 30 05:22:49.271140 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:49.271102    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:49.481735 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:49.481647    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:50.482607 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:50.482556    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:50.891480 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:50.891446    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1d94e3b-c651-4989-afbe-53acbd131f71" path="/var/lib/kubelet/pods/e1d94e3b-c651-4989-afbe-53acbd131f71/volumes"
Oct 30 05:22:52.166934 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:52.166864    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:52.273656 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:52.273611    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:52.380522 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:52.380490    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:53.096602 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:53.096564    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:53.201140 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:53.201112    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:53.894751 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:53.894718    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:54.109809 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:54.109774    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:54.286573 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:54.286487    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:55.229840 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:55.229812    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:55.327011 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:55.326965    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:55.576273 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:55.576222    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w"]
Oct 30 05:22:55.576843 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:55.576804    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w" podUID="8a999b0b-fe75-49de-af7f-bdc3145966c7" containerName="istio-proxy" containerID="cri-o://8a4efacddfe6b2ef605af488de8353636890bc5064c7a6ecb722164682c61bb5" gracePeriod=30
Oct 30 05:22:55.624951 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:55.624914    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:55.744989 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:55.744955    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:55.838030 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:55.837945    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:56.453336 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:56.453306    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:22:56.820283 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:22:56.820189    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:22:57.513926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:57.513881    5898 patch_prober.go:29] interesting pod/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 05:22:57.514355 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:57.513953    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w" podUID="8a999b0b-fe75-49de-af7f-bdc3145966c7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 05:22:58.357567 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:22:58.357539    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:23:00.191081 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:00.191036    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:23:00.858369 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:00.858291    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w"
Oct 30 05:23:01.033206 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:01.033173    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a999b0b-fe75-49de-af7f-bdc3145966c7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a999b0b-fe75-49de-af7f-bdc3145966c7-istio-envoy\") pod \"8a999b0b-fe75-49de-af7f-bdc3145966c7\" (UID: \"8a999b0b-fe75-49de-af7f-bdc3145966c7\") "
Oct 30 05:23:01.033386 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:01.033227    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8a999b0b-fe75-49de-af7f-bdc3145966c7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a999b0b-fe75-49de-af7f-bdc3145966c7-istio-podinfo\") pod \"8a999b0b-fe75-49de-af7f-bdc3145966c7\" (UID: \"8a999b0b-fe75-49de-af7f-bdc3145966c7\") "
Oct 30 05:23:01.033386 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:01.033283    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a999b0b-fe75-49de-af7f-bdc3145966c7-istio-token\" (UniqueName: \"kubernetes.io/projected/8a999b0b-fe75-49de-af7f-bdc3145966c7-istio-token\") pod \"8a999b0b-fe75-49de-af7f-bdc3145966c7\" (UID: \"8a999b0b-fe75-49de-af7f-bdc3145966c7\") "
Oct 30 05:23:01.033386 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:01.033372    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a999b0b-fe75-49de-af7f-bdc3145966c7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a999b0b-fe75-49de-af7f-bdc3145966c7-workload-certs\") pod \"8a999b0b-fe75-49de-af7f-bdc3145966c7\" (UID: \"8a999b0b-fe75-49de-af7f-bdc3145966c7\") "
Oct 30 05:23:01.033536 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:01.033396    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a999b0b-fe75-49de-af7f-bdc3145966c7-kube-api-access-lzwtw\" (UniqueName: \"kubernetes.io/projected/8a999b0b-fe75-49de-af7f-bdc3145966c7-kube-api-access-lzwtw\") pod \"8a999b0b-fe75-49de-af7f-bdc3145966c7\" (UID: \"8a999b0b-fe75-49de-af7f-bdc3145966c7\") "
Oct 30 05:23:01.033536 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:01.033415    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a999b0b-fe75-49de-af7f-bdc3145966c7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a999b0b-fe75-49de-af7f-bdc3145966c7-istiod-ca-cert\") pod \"8a999b0b-fe75-49de-af7f-bdc3145966c7\" (UID: \"8a999b0b-fe75-49de-af7f-bdc3145966c7\") "
Oct 30 05:23:01.033536 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:01.033498    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a999b0b-fe75-49de-af7f-bdc3145966c7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a999b0b-fe75-49de-af7f-bdc3145966c7-workload-socket\") pod \"8a999b0b-fe75-49de-af7f-bdc3145966c7\" (UID: \"8a999b0b-fe75-49de-af7f-bdc3145966c7\") "
Oct 30 05:23:01.033536 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:01.033530    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a999b0b-fe75-49de-af7f-bdc3145966c7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a999b0b-fe75-49de-af7f-bdc3145966c7-credential-socket\") pod \"8a999b0b-fe75-49de-af7f-bdc3145966c7\" (UID: \"8a999b0b-fe75-49de-af7f-bdc3145966c7\") "
Oct 30 05:23:01.034166 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:01.033628    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a999b0b-fe75-49de-af7f-bdc3145966c7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a999b0b-fe75-49de-af7f-bdc3145966c7-istio-data\") pod \"8a999b0b-fe75-49de-af7f-bdc3145966c7\" (UID: \"8a999b0b-fe75-49de-af7f-bdc3145966c7\") "
Oct 30 05:23:01.034166 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:01.033739    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a999b0b-fe75-49de-af7f-bdc3145966c7-workload-certs" pod "8a999b0b-fe75-49de-af7f-bdc3145966c7" (UID: "8a999b0b-fe75-49de-af7f-bdc3145966c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:23:01.034166 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:01.033841    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a999b0b-fe75-49de-af7f-bdc3145966c7-workload-socket" pod "8a999b0b-fe75-49de-af7f-bdc3145966c7" (UID: "8a999b0b-fe75-49de-af7f-bdc3145966c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:23:01.034166 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:01.033992    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a999b0b-fe75-49de-af7f-bdc3145966c7-credential-socket" pod "8a999b0b-fe75-49de-af7f-bdc3145966c7" (UID: "8a999b0b-fe75-49de-af7f-bdc3145966c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:23:01.034166 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:01.034070    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a999b0b-fe75-49de-af7f-bdc3145966c7-istio-data" pod "8a999b0b-fe75-49de-af7f-bdc3145966c7" (UID: "8a999b0b-fe75-49de-af7f-bdc3145966c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:23:01.034434 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:01.034295    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a999b0b-fe75-49de-af7f-bdc3145966c7-istiod-ca-cert" pod "8a999b0b-fe75-49de-af7f-bdc3145966c7" (UID: "8a999b0b-fe75-49de-af7f-bdc3145966c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:23:01.034434 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:01.034378    5898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a999b0b-fe75-49de-af7f-bdc3145966c7-workload-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:23:01.034434 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:01.034393    5898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a999b0b-fe75-49de-af7f-bdc3145966c7-workload-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:23:01.034434 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:01.034401    5898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a999b0b-fe75-49de-af7f-bdc3145966c7-credential-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:23:01.034434 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:01.034411    5898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a999b0b-fe75-49de-af7f-bdc3145966c7-istio-data\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:23:01.036167 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:01.036145    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a999b0b-fe75-49de-af7f-bdc3145966c7-istio-podinfo" pod "8a999b0b-fe75-49de-af7f-bdc3145966c7" (UID: "8a999b0b-fe75-49de-af7f-bdc3145966c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:23:01.036167 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:01.036159    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a999b0b-fe75-49de-af7f-bdc3145966c7-istio-envoy" pod "8a999b0b-fe75-49de-af7f-bdc3145966c7" (UID: "8a999b0b-fe75-49de-af7f-bdc3145966c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:23:01.036308 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:01.036216    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a999b0b-fe75-49de-af7f-bdc3145966c7-istio-token" pod "8a999b0b-fe75-49de-af7f-bdc3145966c7" (UID: "8a999b0b-fe75-49de-af7f-bdc3145966c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:23:01.036308 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:01.036246    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a999b0b-fe75-49de-af7f-bdc3145966c7-kube-api-access-lzwtw" pod "8a999b0b-fe75-49de-af7f-bdc3145966c7" (UID: "8a999b0b-fe75-49de-af7f-bdc3145966c7"). InnerVolumeSpecName "kube-api-access-lzwtw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:23:01.136379 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:01.136295    5898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a999b0b-fe75-49de-af7f-bdc3145966c7-istio-envoy\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:23:01.136379 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:01.136330    5898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a999b0b-fe75-49de-af7f-bdc3145966c7-istio-podinfo\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:23:01.136379 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:01.136340    5898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a999b0b-fe75-49de-af7f-bdc3145966c7-istio-token\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:23:01.136379 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:01.136348    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzwtw\" (UniqueName: \"kubernetes.io/projected/8a999b0b-fe75-49de-af7f-bdc3145966c7-kube-api-access-lzwtw\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:23:01.136379 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:01.136359    5898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a999b0b-fe75-49de-af7f-bdc3145966c7-istiod-ca-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:23:01.186046 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:01.186015    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:23:01.243293 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:01.243254    5898 generic.go:356] "Generic (PLEG): container finished" podID="8a999b0b-fe75-49de-af7f-bdc3145966c7" containerID="8a4efacddfe6b2ef605af488de8353636890bc5064c7a6ecb722164682c61bb5" exitCode=0
Oct 30 05:23:01.243782 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:01.243313    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w"
Oct 30 05:23:01.243782 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:01.243341    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w" event={"ID":"8a999b0b-fe75-49de-af7f-bdc3145966c7","Type":"ContainerDied","Data":"8a4efacddfe6b2ef605af488de8353636890bc5064c7a6ecb722164682c61bb5"}
Oct 30 05:23:01.243782 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:01.243413    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w" event={"ID":"8a999b0b-fe75-49de-af7f-bdc3145966c7","Type":"ContainerDied","Data":"495031a0f11906c630cb64d5f4a829aee63dbb2bed624b486eb2a1742e8ecab0"}
Oct 30 05:23:01.243782 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:01.243448    5898 scope.go:122] "RemoveContainer" containerID="8a4efacddfe6b2ef605af488de8353636890bc5064c7a6ecb722164682c61bb5"
Oct 30 05:23:01.272007 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:01.271977    5898 scope.go:122] "RemoveContainer" containerID="59e7172c0e4cddcdc33c748a06b25fba80835c7d5e4cd741ef64bc5dfe9b03e5"
Oct 30 05:23:01.284333 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:01.284313    5898 scope.go:122] "RemoveContainer" containerID="8a4efacddfe6b2ef605af488de8353636890bc5064c7a6ecb722164682c61bb5"
Oct 30 05:23:01.284664 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:23:01.284643    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a4efacddfe6b2ef605af488de8353636890bc5064c7a6ecb722164682c61bb5\": container with ID starting with 8a4efacddfe6b2ef605af488de8353636890bc5064c7a6ecb722164682c61bb5 not found: ID does not exist" containerID="8a4efacddfe6b2ef605af488de8353636890bc5064c7a6ecb722164682c61bb5"
Oct 30 05:23:01.284726 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:01.284673    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a4efacddfe6b2ef605af488de8353636890bc5064c7a6ecb722164682c61bb5"} err="failed to get container status \"8a4efacddfe6b2ef605af488de8353636890bc5064c7a6ecb722164682c61bb5\": rpc error: code = NotFound desc = could not find container \"8a4efacddfe6b2ef605af488de8353636890bc5064c7a6ecb722164682c61bb5\": container with ID starting with 8a4efacddfe6b2ef605af488de8353636890bc5064c7a6ecb722164682c61bb5 not found: ID does not exist"
Oct 30 05:23:01.284726 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:01.284691    5898 scope.go:122] "RemoveContainer" containerID="59e7172c0e4cddcdc33c748a06b25fba80835c7d5e4cd741ef64bc5dfe9b03e5"
Oct 30 05:23:01.284967 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:23:01.284951    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59e7172c0e4cddcdc33c748a06b25fba80835c7d5e4cd741ef64bc5dfe9b03e5\": container with ID starting with 59e7172c0e4cddcdc33c748a06b25fba80835c7d5e4cd741ef64bc5dfe9b03e5 not found: ID does not exist" containerID="59e7172c0e4cddcdc33c748a06b25fba80835c7d5e4cd741ef64bc5dfe9b03e5"
Oct 30 05:23:01.285023 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:01.284972    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59e7172c0e4cddcdc33c748a06b25fba80835c7d5e4cd741ef64bc5dfe9b03e5"} err="failed to get container status \"59e7172c0e4cddcdc33c748a06b25fba80835c7d5e4cd741ef64bc5dfe9b03e5\": rpc error: code = NotFound desc = could not find container \"59e7172c0e4cddcdc33c748a06b25fba80835c7d5e4cd741ef64bc5dfe9b03e5\": container with ID starting with 59e7172c0e4cddcdc33c748a06b25fba80835c7d5e4cd741ef64bc5dfe9b03e5 not found: ID does not exist"
Oct 30 05:23:01.308857 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:01.308834    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w"]
Oct 30 05:23:01.319798 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:01.319777    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-356cae-openshift-default-5bd47d6d5-mkk9w"]
Oct 30 05:23:02.088785 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:02.088752    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:23:02.893857 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:02.893820    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a999b0b-fe75-49de-af7f-bdc3145966c7" path="/var/lib/kubelet/pods/8a999b0b-fe75-49de-af7f-bdc3145966c7/volumes"
Oct 30 05:23:03.822927 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:03.811510    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:23:05.965599 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:05.965566    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:23:06.866141 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:06.866096    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:23:06.950670 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:06.950636    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:23:08.018214 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:08.018182    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:23:09.885629 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:09.885589    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:23:11.677449 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:11.677416    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:23:13.387218 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:13.387178    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:23:14.974087 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:14.974050    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:23:15.074180 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:15.074140    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:23:15.190818 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:15.190781    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:23:16.982543 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:16.982508    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:23:18.722935 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:18.722858    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:23:20.479365 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:20.479311    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:23:22.184957 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:22.184893    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:23:22.278677 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:22.278638    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:23:24.071644 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:24.071608    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:23:25.731704 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:25.731669    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:23:25.842083 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:25.841523    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:23:26.548994 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:26.548962    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:23:26.820018 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:23:26.819934    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:23:27.659354 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:27.659319    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:23:28.986102 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:28.986070    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242"]
Oct 30 05:23:28.986662 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:28.986643    5898 cpu_manager.go:414] "containerMap: removing container" podUID="ee5dfade-6b22-4653-80d9-75320b4c6117" containerName="main"
Oct 30 05:23:28.986662 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:28.986660    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee5dfade-6b22-4653-80d9-75320b4c6117" containerName="main"
Oct 30 05:23:28.986837 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:28.986670    5898 cpu_manager.go:414] "containerMap: removing container" podUID="e1d94e3b-c651-4989-afbe-53acbd131f71" containerName="istio-proxy"
Oct 30 05:23:28.986837 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:28.986676    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1d94e3b-c651-4989-afbe-53acbd131f71" containerName="istio-proxy"
Oct 30 05:23:28.986837 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:28.986688    5898 cpu_manager.go:414] "containerMap: removing container" podUID="96952584-fdf3-4731-a845-83ec28cd69b4" containerName="maas-api"
Oct 30 05:23:28.986837 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:28.986693    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96952584-fdf3-4731-a845-83ec28cd69b4" containerName="maas-api"
Oct 30 05:23:28.986837 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:28.986709    5898 cpu_manager.go:414] "containerMap: removing container" podUID="8a999b0b-fe75-49de-af7f-bdc3145966c7" containerName="istio-proxy"
Oct 30 05:23:28.986837 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:28.986716    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a999b0b-fe75-49de-af7f-bdc3145966c7" containerName="istio-proxy"
Oct 30 05:23:28.986837 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:28.986736    5898 cpu_manager.go:414] "containerMap: removing container" podUID="4d4f8e87-1d0e-434a-9be6-d66558d209eb" containerName="main"
Oct 30 05:23:28.986837 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:28.986741    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d4f8e87-1d0e-434a-9be6-d66558d209eb" containerName="main"
Oct 30 05:23:28.986837 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:28.986752    5898 cpu_manager.go:414] "containerMap: removing container" podUID="8a999b0b-fe75-49de-af7f-bdc3145966c7" containerName="istio-proxy"
Oct 30 05:23:28.986837 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:28.986757    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a999b0b-fe75-49de-af7f-bdc3145966c7" containerName="istio-proxy"
Oct 30 05:23:28.986837 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:28.986778    5898 cpu_manager.go:414] "containerMap: removing container" podUID="e1d94e3b-c651-4989-afbe-53acbd131f71" containerName="istio-proxy"
Oct 30 05:23:28.986837 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:28.986786    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1d94e3b-c651-4989-afbe-53acbd131f71" containerName="istio-proxy"
Oct 30 05:23:28.987402 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:28.986879    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee5dfade-6b22-4653-80d9-75320b4c6117" containerName="main"
Oct 30 05:23:28.987402 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:28.986890    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1d94e3b-c651-4989-afbe-53acbd131f71" containerName="istio-proxy"
Oct 30 05:23:28.987402 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:28.986896    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="96952584-fdf3-4731-a845-83ec28cd69b4" containerName="maas-api"
Oct 30 05:23:28.987402 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:28.986917    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a999b0b-fe75-49de-af7f-bdc3145966c7" containerName="istio-proxy"
Oct 30 05:23:28.987402 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:28.986923    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d4f8e87-1d0e-434a-9be6-d66558d209eb" containerName="main"
Oct 30 05:23:28.987402 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:28.986928    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a999b0b-fe75-49de-af7f-bdc3145966c7" containerName="istio-proxy"
Oct 30 05:23:28.990335 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:28.990317    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242"
Oct 30 05:23:29.004304 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:29.004283    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-68b893-openshift-default-dockercfg-6ns6m\""
Oct 30 05:23:29.018999 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:29.018976    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242"]
Oct 30 05:23:29.044044 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:29.044010    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5df41ffb-123b-45fd-a3ae-3c61bcaafd5a-credential-socket\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242\" (UID: \"5df41ffb-123b-45fd-a3ae-3c61bcaafd5a\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242"
Oct 30 05:23:29.044177 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:29.044061    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fdqm\" (UniqueName: \"kubernetes.io/projected/5df41ffb-123b-45fd-a3ae-3c61bcaafd5a-kube-api-access-4fdqm\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242\" (UID: \"5df41ffb-123b-45fd-a3ae-3c61bcaafd5a\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242"
Oct 30 05:23:29.044177 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:29.044164    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5df41ffb-123b-45fd-a3ae-3c61bcaafd5a-workload-certs\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242\" (UID: \"5df41ffb-123b-45fd-a3ae-3c61bcaafd5a\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242"
Oct 30 05:23:29.044291 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:29.044214    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5df41ffb-123b-45fd-a3ae-3c61bcaafd5a-workload-socket\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242\" (UID: \"5df41ffb-123b-45fd-a3ae-3c61bcaafd5a\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242"
Oct 30 05:23:29.044399 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:29.044374    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5df41ffb-123b-45fd-a3ae-3c61bcaafd5a-istiod-ca-cert\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242\" (UID: \"5df41ffb-123b-45fd-a3ae-3c61bcaafd5a\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242"
Oct 30 05:23:29.044453 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:29.044420    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5df41ffb-123b-45fd-a3ae-3c61bcaafd5a-istio-token\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242\" (UID: \"5df41ffb-123b-45fd-a3ae-3c61bcaafd5a\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242"
Oct 30 05:23:29.044616 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:29.044598    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5df41ffb-123b-45fd-a3ae-3c61bcaafd5a-istio-envoy\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242\" (UID: \"5df41ffb-123b-45fd-a3ae-3c61bcaafd5a\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242"
Oct 30 05:23:29.044657 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:29.044638    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5df41ffb-123b-45fd-a3ae-3c61bcaafd5a-istio-podinfo\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242\" (UID: \"5df41ffb-123b-45fd-a3ae-3c61bcaafd5a\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242"
Oct 30 05:23:29.044713 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:29.044698    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5df41ffb-123b-45fd-a3ae-3c61bcaafd5a-istio-data\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242\" (UID: \"5df41ffb-123b-45fd-a3ae-3c61bcaafd5a\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242"
Oct 30 05:23:29.146183 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:29.146144    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5df41ffb-123b-45fd-a3ae-3c61bcaafd5a-istio-data\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242\" (UID: \"5df41ffb-123b-45fd-a3ae-3c61bcaafd5a\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242"
Oct 30 05:23:29.146365 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:29.146243    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5df41ffb-123b-45fd-a3ae-3c61bcaafd5a-credential-socket\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242\" (UID: \"5df41ffb-123b-45fd-a3ae-3c61bcaafd5a\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242"
Oct 30 05:23:29.146365 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:29.146277    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fdqm\" (UniqueName: \"kubernetes.io/projected/5df41ffb-123b-45fd-a3ae-3c61bcaafd5a-kube-api-access-4fdqm\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242\" (UID: \"5df41ffb-123b-45fd-a3ae-3c61bcaafd5a\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242"
Oct 30 05:23:29.146365 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:29.146317    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5df41ffb-123b-45fd-a3ae-3c61bcaafd5a-workload-certs\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242\" (UID: \"5df41ffb-123b-45fd-a3ae-3c61bcaafd5a\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242"
Oct 30 05:23:29.146365 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:29.146357    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5df41ffb-123b-45fd-a3ae-3c61bcaafd5a-workload-socket\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242\" (UID: \"5df41ffb-123b-45fd-a3ae-3c61bcaafd5a\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242"
Oct 30 05:23:29.146636 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:29.146433    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5df41ffb-123b-45fd-a3ae-3c61bcaafd5a-istiod-ca-cert\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242\" (UID: \"5df41ffb-123b-45fd-a3ae-3c61bcaafd5a\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242"
Oct 30 05:23:29.146636 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:29.146464    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5df41ffb-123b-45fd-a3ae-3c61bcaafd5a-istio-token\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242\" (UID: \"5df41ffb-123b-45fd-a3ae-3c61bcaafd5a\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242"
Oct 30 05:23:29.146636 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:29.146597    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5df41ffb-123b-45fd-a3ae-3c61bcaafd5a-istio-envoy\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242\" (UID: \"5df41ffb-123b-45fd-a3ae-3c61bcaafd5a\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242"
Oct 30 05:23:29.146636 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:29.146629    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5df41ffb-123b-45fd-a3ae-3c61bcaafd5a-istio-podinfo\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242\" (UID: \"5df41ffb-123b-45fd-a3ae-3c61bcaafd5a\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242"
Oct 30 05:23:29.146887 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:29.146861    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5df41ffb-123b-45fd-a3ae-3c61bcaafd5a-workload-socket\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242\" (UID: \"5df41ffb-123b-45fd-a3ae-3c61bcaafd5a\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242"
Oct 30 05:23:29.147547 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:29.147503    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5df41ffb-123b-45fd-a3ae-3c61bcaafd5a-istio-data\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242\" (UID: \"5df41ffb-123b-45fd-a3ae-3c61bcaafd5a\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242"
Oct 30 05:23:29.147748 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:29.147652    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5df41ffb-123b-45fd-a3ae-3c61bcaafd5a-istiod-ca-cert\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242\" (UID: \"5df41ffb-123b-45fd-a3ae-3c61bcaafd5a\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242"
Oct 30 05:23:29.147865 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:29.147669    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5df41ffb-123b-45fd-a3ae-3c61bcaafd5a-credential-socket\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242\" (UID: \"5df41ffb-123b-45fd-a3ae-3c61bcaafd5a\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242"
Oct 30 05:23:29.147865 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:29.147692    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5df41ffb-123b-45fd-a3ae-3c61bcaafd5a-workload-certs\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242\" (UID: \"5df41ffb-123b-45fd-a3ae-3c61bcaafd5a\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242"
Oct 30 05:23:29.149706 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:29.149678    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5df41ffb-123b-45fd-a3ae-3c61bcaafd5a-istio-envoy\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242\" (UID: \"5df41ffb-123b-45fd-a3ae-3c61bcaafd5a\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242"
Oct 30 05:23:29.149828 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:29.149811    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5df41ffb-123b-45fd-a3ae-3c61bcaafd5a-istio-podinfo\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242\" (UID: \"5df41ffb-123b-45fd-a3ae-3c61bcaafd5a\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242"
Oct 30 05:23:29.165480 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:29.165450    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fdqm\" (UniqueName: \"kubernetes.io/projected/5df41ffb-123b-45fd-a3ae-3c61bcaafd5a-kube-api-access-4fdqm\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242\" (UID: \"5df41ffb-123b-45fd-a3ae-3c61bcaafd5a\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242"
Oct 30 05:23:29.165585 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:29.165534    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5df41ffb-123b-45fd-a3ae-3c61bcaafd5a-istio-token\") pod \"e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242\" (UID: \"5df41ffb-123b-45fd-a3ae-3c61bcaafd5a\") " pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242"
Oct 30 05:23:29.303989 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:29.303884    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242"
Oct 30 05:23:29.436043 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:29.435996    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:23:29.620829 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:29.620798    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242"]
Oct 30 05:23:29.623087 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:23:29.623058    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5df41ffb_123b_45fd_a3ae_3c61bcaafd5a.slice/crio-89b92e881b70344371b2e3c371822626cad4379c79e0b92f2ab1e9c3805f2579 WatchSource:0}: Error finding container 89b92e881b70344371b2e3c371822626cad4379c79e0b92f2ab1e9c3805f2579: Status 404 returned error can't find the container with id 89b92e881b70344371b2e3c371822626cad4379c79e0b92f2ab1e9c3805f2579
Oct 30 05:23:29.625796 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:29.625760    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:23:29.625938 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:29.625834    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:23:29.625938 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:29.625864    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:23:30.506149 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:30.506108    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242" event={"ID":"5df41ffb-123b-45fd-a3ae-3c61bcaafd5a","Type":"ContainerStarted","Data":"8432784f5544c48a0c29699cdeecfd6a240a51c9f80e367d5bb3995bc48bde10"}
Oct 30 05:23:30.506548 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:30.506168    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242" event={"ID":"5df41ffb-123b-45fd-a3ae-3c61bcaafd5a","Type":"ContainerStarted","Data":"89b92e881b70344371b2e3c371822626cad4379c79e0b92f2ab1e9c3805f2579"}
Oct 30 05:23:30.567284 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:30.567230    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242" podStartSLOduration=2.567214688 podStartE2EDuration="2.567214688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:23:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:23:30.55517058 +0000 UTC m=+2339.778990460" watchObservedRunningTime="2026-10-30 05:23:30.567214688 +0000 UTC m=+2339.791034589"
Oct 30 05:23:31.256053 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:31.256018    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:23:31.307920 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:31.307358    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242"
Oct 30 05:23:31.313925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:31.313628    5898 patch_prober.go:29] interesting pod/e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Oct 30 05:23:31.313925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:31.313691    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242" podUID="5df41ffb-123b-45fd-a3ae-3c61bcaafd5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Oct 30 05:23:32.304529 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:32.304482    5898 patch_prober.go:29] interesting pod/e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Oct 30 05:23:32.305037 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:32.304542    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242" podUID="5df41ffb-123b-45fd-a3ae-3c61bcaafd5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Oct 30 05:23:33.030936 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:33.030828    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:23:33.131297 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:33.131265    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:23:33.304492 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:33.304406    5898 patch_prober.go:29] interesting pod/e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Oct 30 05:23:33.304492 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:33.304469    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242" podUID="5df41ffb-123b-45fd-a3ae-3c61bcaafd5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Oct 30 05:23:34.305160 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:34.305124    5898 patch_prober.go:29] interesting pod/e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Oct 30 05:23:34.306144 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:34.306111    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242" podUID="5df41ffb-123b-45fd-a3ae-3c61bcaafd5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Oct 30 05:23:34.873774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:34.873730    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:23:34.951577 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:34.951539    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:23:35.052729 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:35.052697    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:23:35.304612 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:35.304519    5898 patch_prober.go:29] interesting pod/e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Oct 30 05:23:35.304612 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:35.304586    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242" podUID="5df41ffb-123b-45fd-a3ae-3c61bcaafd5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Oct 30 05:23:36.318016 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:36.317985    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242"
Oct 30 05:23:36.319928 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:36.319885    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242"
Oct 30 05:23:36.321303 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:36.321282    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242"
Oct 30 05:23:36.757057 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:36.757021    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:23:38.661641 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:38.661607    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:23:40.353460 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:40.353427    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:23:41.054032 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:41.053985    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:23:41.940646 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:41.940610    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6"]
Oct 30 05:23:41.941091 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:41.940820    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6" podUID="79faca01-0058-4ef5-ab44-5bb52d1c3bb0" containerName="istio-proxy" containerID="cri-o://d0d299da8d8a4c1b0efe4f1aabee42d6b5f15a6fec08fd012bed4af08c1081a2" gracePeriod=30
Oct 30 05:23:42.068753 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:42.068662    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:23:42.169940 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:42.169560    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:23:43.858831 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:43.858800    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:23:44.897915 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:44.897867    5898 patch_prober.go:29] interesting pod/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 05:23:44.898430 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:44.897976    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6" podUID="79faca01-0058-4ef5-ab44-5bb52d1c3bb0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 05:23:45.342846 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:45.342808    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:23:46.006819 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:46.006768    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:23:46.113880 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:46.113837    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:23:46.197277 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:46.197243    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:23:46.643141 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:46.643110    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:23:47.219861 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:47.219797    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6"
Oct 30 05:23:47.258427 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:47.258400    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79faca01-0058-4ef5-ab44-5bb52d1c3bb0-istio-token\" (UniqueName: \"kubernetes.io/projected/79faca01-0058-4ef5-ab44-5bb52d1c3bb0-istio-token\") pod \"79faca01-0058-4ef5-ab44-5bb52d1c3bb0\" (UID: \"79faca01-0058-4ef5-ab44-5bb52d1c3bb0\") "
Oct 30 05:23:47.258587 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:47.258518    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79faca01-0058-4ef5-ab44-5bb52d1c3bb0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79faca01-0058-4ef5-ab44-5bb52d1c3bb0-istio-envoy\") pod \"79faca01-0058-4ef5-ab44-5bb52d1c3bb0\" (UID: \"79faca01-0058-4ef5-ab44-5bb52d1c3bb0\") "
Oct 30 05:23:47.258587 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:47.258571    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79faca01-0058-4ef5-ab44-5bb52d1c3bb0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79faca01-0058-4ef5-ab44-5bb52d1c3bb0-istio-podinfo\") pod \"79faca01-0058-4ef5-ab44-5bb52d1c3bb0\" (UID: \"79faca01-0058-4ef5-ab44-5bb52d1c3bb0\") "
Oct 30 05:23:47.258713 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:47.258676    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79faca01-0058-4ef5-ab44-5bb52d1c3bb0-kube-api-access-w95rq\" (UniqueName: \"kubernetes.io/projected/79faca01-0058-4ef5-ab44-5bb52d1c3bb0-kube-api-access-w95rq\") pod \"79faca01-0058-4ef5-ab44-5bb52d1c3bb0\" (UID: \"79faca01-0058-4ef5-ab44-5bb52d1c3bb0\") "
Oct 30 05:23:47.258825 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:47.258790    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79faca01-0058-4ef5-ab44-5bb52d1c3bb0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79faca01-0058-4ef5-ab44-5bb52d1c3bb0-workload-certs\") pod \"79faca01-0058-4ef5-ab44-5bb52d1c3bb0\" (UID: \"79faca01-0058-4ef5-ab44-5bb52d1c3bb0\") "
Oct 30 05:23:47.258891 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:47.258852    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79faca01-0058-4ef5-ab44-5bb52d1c3bb0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79faca01-0058-4ef5-ab44-5bb52d1c3bb0-workload-socket\") pod \"79faca01-0058-4ef5-ab44-5bb52d1c3bb0\" (UID: \"79faca01-0058-4ef5-ab44-5bb52d1c3bb0\") "
Oct 30 05:23:47.258979 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:47.258889    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79faca01-0058-4ef5-ab44-5bb52d1c3bb0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79faca01-0058-4ef5-ab44-5bb52d1c3bb0-credential-socket\") pod \"79faca01-0058-4ef5-ab44-5bb52d1c3bb0\" (UID: \"79faca01-0058-4ef5-ab44-5bb52d1c3bb0\") "
Oct 30 05:23:47.259059 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:47.258972    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79faca01-0058-4ef5-ab44-5bb52d1c3bb0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79faca01-0058-4ef5-ab44-5bb52d1c3bb0-istiod-ca-cert\") pod \"79faca01-0058-4ef5-ab44-5bb52d1c3bb0\" (UID: \"79faca01-0058-4ef5-ab44-5bb52d1c3bb0\") "
Oct 30 05:23:47.259059 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:47.259023    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79faca01-0058-4ef5-ab44-5bb52d1c3bb0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79faca01-0058-4ef5-ab44-5bb52d1c3bb0-istio-data\") pod \"79faca01-0058-4ef5-ab44-5bb52d1c3bb0\" (UID: \"79faca01-0058-4ef5-ab44-5bb52d1c3bb0\") "
Oct 30 05:23:47.259270 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:47.259241    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79faca01-0058-4ef5-ab44-5bb52d1c3bb0-workload-certs" pod "79faca01-0058-4ef5-ab44-5bb52d1c3bb0" (UID: "79faca01-0058-4ef5-ab44-5bb52d1c3bb0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:23:47.259488 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:47.259462    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79faca01-0058-4ef5-ab44-5bb52d1c3bb0-istio-data" pod "79faca01-0058-4ef5-ab44-5bb52d1c3bb0" (UID: "79faca01-0058-4ef5-ab44-5bb52d1c3bb0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:23:47.259689 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:47.259667    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79faca01-0058-4ef5-ab44-5bb52d1c3bb0-workload-socket" pod "79faca01-0058-4ef5-ab44-5bb52d1c3bb0" (UID: "79faca01-0058-4ef5-ab44-5bb52d1c3bb0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:23:47.259873 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:47.259853    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79faca01-0058-4ef5-ab44-5bb52d1c3bb0-credential-socket" pod "79faca01-0058-4ef5-ab44-5bb52d1c3bb0" (UID: "79faca01-0058-4ef5-ab44-5bb52d1c3bb0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:23:47.260016 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:47.259928    5898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79faca01-0058-4ef5-ab44-5bb52d1c3bb0-workload-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:23:47.260016 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:47.259955    5898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79faca01-0058-4ef5-ab44-5bb52d1c3bb0-istio-data\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:23:47.260177 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:47.260037    5898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79faca01-0058-4ef5-ab44-5bb52d1c3bb0-workload-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:23:47.260397 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:47.260372    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79faca01-0058-4ef5-ab44-5bb52d1c3bb0-istiod-ca-cert" pod "79faca01-0058-4ef5-ab44-5bb52d1c3bb0" (UID: "79faca01-0058-4ef5-ab44-5bb52d1c3bb0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:23:47.261088 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:47.261068    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79faca01-0058-4ef5-ab44-5bb52d1c3bb0-istio-envoy" pod "79faca01-0058-4ef5-ab44-5bb52d1c3bb0" (UID: "79faca01-0058-4ef5-ab44-5bb52d1c3bb0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:23:47.261469 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:47.261451    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79faca01-0058-4ef5-ab44-5bb52d1c3bb0-istio-podinfo" pod "79faca01-0058-4ef5-ab44-5bb52d1c3bb0" (UID: "79faca01-0058-4ef5-ab44-5bb52d1c3bb0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:23:47.262350 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:47.262322    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79faca01-0058-4ef5-ab44-5bb52d1c3bb0-kube-api-access-w95rq" pod "79faca01-0058-4ef5-ab44-5bb52d1c3bb0" (UID: "79faca01-0058-4ef5-ab44-5bb52d1c3bb0"). InnerVolumeSpecName "kube-api-access-w95rq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:23:47.263649 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:47.263630    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79faca01-0058-4ef5-ab44-5bb52d1c3bb0-istio-token" pod "79faca01-0058-4ef5-ab44-5bb52d1c3bb0" (UID: "79faca01-0058-4ef5-ab44-5bb52d1c3bb0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:23:47.360840 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:47.360808    5898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79faca01-0058-4ef5-ab44-5bb52d1c3bb0-istio-envoy\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:23:47.360840 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:47.360835    5898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79faca01-0058-4ef5-ab44-5bb52d1c3bb0-istio-podinfo\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:23:47.360840 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:47.360845    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w95rq\" (UniqueName: \"kubernetes.io/projected/79faca01-0058-4ef5-ab44-5bb52d1c3bb0-kube-api-access-w95rq\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:23:47.361107 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:47.360855    5898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79faca01-0058-4ef5-ab44-5bb52d1c3bb0-credential-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:23:47.361107 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:47.360864    5898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79faca01-0058-4ef5-ab44-5bb52d1c3bb0-istiod-ca-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:23:47.361107 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:47.360875    5898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79faca01-0058-4ef5-ab44-5bb52d1c3bb0-istio-token\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:23:47.655567 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:47.655519    5898 generic.go:356] "Generic (PLEG): container finished" podID="79faca01-0058-4ef5-ab44-5bb52d1c3bb0" containerID="d0d299da8d8a4c1b0efe4f1aabee42d6b5f15a6fec08fd012bed4af08c1081a2" exitCode=0
Oct 30 05:23:47.655748 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:47.655591    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6"
Oct 30 05:23:47.655748 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:47.655602    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6" event={"ID":"79faca01-0058-4ef5-ab44-5bb52d1c3bb0","Type":"ContainerDied","Data":"d0d299da8d8a4c1b0efe4f1aabee42d6b5f15a6fec08fd012bed4af08c1081a2"}
Oct 30 05:23:47.655748 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:47.655693    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6" event={"ID":"79faca01-0058-4ef5-ab44-5bb52d1c3bb0","Type":"ContainerDied","Data":"dde0f1b0ca7236bc54e4bf21e31acbd27aa29ea937c9507645e8ee3dda57e55e"}
Oct 30 05:23:47.655748 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:47.655718    5898 scope.go:122] "RemoveContainer" containerID="d0d299da8d8a4c1b0efe4f1aabee42d6b5f15a6fec08fd012bed4af08c1081a2"
Oct 30 05:23:47.667908 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:47.667881    5898 scope.go:122] "RemoveContainer" containerID="d0d299da8d8a4c1b0efe4f1aabee42d6b5f15a6fec08fd012bed4af08c1081a2"
Oct 30 05:23:47.668182 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:23:47.668160    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0d299da8d8a4c1b0efe4f1aabee42d6b5f15a6fec08fd012bed4af08c1081a2\": container with ID starting with d0d299da8d8a4c1b0efe4f1aabee42d6b5f15a6fec08fd012bed4af08c1081a2 not found: ID does not exist" containerID="d0d299da8d8a4c1b0efe4f1aabee42d6b5f15a6fec08fd012bed4af08c1081a2"
Oct 30 05:23:47.668265 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:47.668194    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0d299da8d8a4c1b0efe4f1aabee42d6b5f15a6fec08fd012bed4af08c1081a2"} err="failed to get container status \"d0d299da8d8a4c1b0efe4f1aabee42d6b5f15a6fec08fd012bed4af08c1081a2\": rpc error: code = NotFound desc = could not find container \"d0d299da8d8a4c1b0efe4f1aabee42d6b5f15a6fec08fd012bed4af08c1081a2\": container with ID starting with d0d299da8d8a4c1b0efe4f1aabee42d6b5f15a6fec08fd012bed4af08c1081a2 not found: ID does not exist"
Oct 30 05:23:47.705454 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:47.705423    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6"]
Oct 30 05:23:47.716656 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:47.716631    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-sl2x6"]
Oct 30 05:23:47.896062 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:47.896027    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:23:48.890972 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:48.890938    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79faca01-0058-4ef5-ab44-5bb52d1c3bb0" path="/var/lib/kubelet/pods/79faca01-0058-4ef5-ab44-5bb52d1c3bb0/volumes"
Oct 30 05:23:49.629061 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:49.629031    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:23:51.444744 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:51.444698    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:23:53.025135 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:53.025050    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:23:53.125603 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:53.125568    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:23:53.227777 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:53.227714    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:23:53.372573 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:53.372543    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:23:54.855202 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:54.855162    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:23:54.959028 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:54.956784    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:23:56.813175 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:56.813128    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:23:56.818948 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:23:56.818880    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:23:58.653961 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:23:58.653931    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:24:00.645895 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:00.645853    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:24:01.785656 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:01.785622    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48"]
Oct 30 05:24:01.786111 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:01.785818    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48" podUID="66f813c3-c7c9-452d-839d-4710108f03c4" containerName="istio-proxy" containerID="cri-o://3f2bbd9c32545fbf42a40c68f01534d2b515e7194ce1c896e7d5e310c3949e34" gracePeriod=30
Oct 30 05:24:02.472503 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:02.472469    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:24:04.246049 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:04.246014    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:24:04.547062 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:04.546984    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm"]
Oct 30 05:24:04.547619 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:04.547602    5898 cpu_manager.go:414] "containerMap: removing container" podUID="79faca01-0058-4ef5-ab44-5bb52d1c3bb0" containerName="istio-proxy"
Oct 30 05:24:04.547667 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:04.547619    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79faca01-0058-4ef5-ab44-5bb52d1c3bb0" containerName="istio-proxy"
Oct 30 05:24:04.547728 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:04.547719    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1d94e3b-c651-4989-afbe-53acbd131f71" containerName="istio-proxy"
Oct 30 05:24:04.547764 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:04.547733    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="79faca01-0058-4ef5-ab44-5bb52d1c3bb0" containerName="istio-proxy"
Oct 30 05:24:04.551451 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:04.551430    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm"
Oct 30 05:24:04.572861 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:04.572831    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm"]
Oct 30 05:24:04.659224 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:04.659194    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-6b3855-openshift-default-dockercfg-vbtxb\""
Oct 30 05:24:04.703930 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:04.703880    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c26b723-62af-41b3-bf3a-36a48065a924-workload-certs\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm\" (UID: \"1c26b723-62af-41b3-bf3a-36a48065a924\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm"
Oct 30 05:24:04.704111 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:04.703950    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c26b723-62af-41b3-bf3a-36a48065a924-istio-podinfo\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm\" (UID: \"1c26b723-62af-41b3-bf3a-36a48065a924\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm"
Oct 30 05:24:04.704111 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:04.703976    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c26b723-62af-41b3-bf3a-36a48065a924-istio-data\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm\" (UID: \"1c26b723-62af-41b3-bf3a-36a48065a924\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm"
Oct 30 05:24:04.704245 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:04.704144    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c26b723-62af-41b3-bf3a-36a48065a924-credential-socket\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm\" (UID: \"1c26b723-62af-41b3-bf3a-36a48065a924\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm"
Oct 30 05:24:04.704245 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:04.704187    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c26b723-62af-41b3-bf3a-36a48065a924-istiod-ca-cert\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm\" (UID: \"1c26b723-62af-41b3-bf3a-36a48065a924\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm"
Oct 30 05:24:04.704357 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:04.704299    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c26b723-62af-41b3-bf3a-36a48065a924-istio-token\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm\" (UID: \"1c26b723-62af-41b3-bf3a-36a48065a924\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm"
Oct 30 05:24:04.704455 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:04.704436    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c26b723-62af-41b3-bf3a-36a48065a924-workload-socket\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm\" (UID: \"1c26b723-62af-41b3-bf3a-36a48065a924\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm"
Oct 30 05:24:04.704523 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:04.704500    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c26b723-62af-41b3-bf3a-36a48065a924-istio-envoy\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm\" (UID: \"1c26b723-62af-41b3-bf3a-36a48065a924\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm"
Oct 30 05:24:04.704563 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:04.704545    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6rcj\" (UniqueName: \"kubernetes.io/projected/1c26b723-62af-41b3-bf3a-36a48065a924-kube-api-access-f6rcj\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm\" (UID: \"1c26b723-62af-41b3-bf3a-36a48065a924\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm"
Oct 30 05:24:04.805413 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:04.805334    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c26b723-62af-41b3-bf3a-36a48065a924-workload-socket\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm\" (UID: \"1c26b723-62af-41b3-bf3a-36a48065a924\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm"
Oct 30 05:24:04.805413 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:04.805371    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c26b723-62af-41b3-bf3a-36a48065a924-istio-envoy\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm\" (UID: \"1c26b723-62af-41b3-bf3a-36a48065a924\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm"
Oct 30 05:24:04.805413 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:04.805399    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6rcj\" (UniqueName: \"kubernetes.io/projected/1c26b723-62af-41b3-bf3a-36a48065a924-kube-api-access-f6rcj\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm\" (UID: \"1c26b723-62af-41b3-bf3a-36a48065a924\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm"
Oct 30 05:24:04.805650 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:04.805440    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c26b723-62af-41b3-bf3a-36a48065a924-workload-certs\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm\" (UID: \"1c26b723-62af-41b3-bf3a-36a48065a924\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm"
Oct 30 05:24:04.805650 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:04.805480    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c26b723-62af-41b3-bf3a-36a48065a924-istio-podinfo\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm\" (UID: \"1c26b723-62af-41b3-bf3a-36a48065a924\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm"
Oct 30 05:24:04.805650 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:04.805511    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c26b723-62af-41b3-bf3a-36a48065a924-istio-data\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm\" (UID: \"1c26b723-62af-41b3-bf3a-36a48065a924\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm"
Oct 30 05:24:04.805650 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:04.805580    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c26b723-62af-41b3-bf3a-36a48065a924-credential-socket\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm\" (UID: \"1c26b723-62af-41b3-bf3a-36a48065a924\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm"
Oct 30 05:24:04.805650 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:04.805612    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c26b723-62af-41b3-bf3a-36a48065a924-istiod-ca-cert\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm\" (UID: \"1c26b723-62af-41b3-bf3a-36a48065a924\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm"
Oct 30 05:24:04.805954 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:04.805667    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c26b723-62af-41b3-bf3a-36a48065a924-istio-token\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm\" (UID: \"1c26b723-62af-41b3-bf3a-36a48065a924\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm"
Oct 30 05:24:04.805954 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:04.805857    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c26b723-62af-41b3-bf3a-36a48065a924-workload-certs\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm\" (UID: \"1c26b723-62af-41b3-bf3a-36a48065a924\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm"
Oct 30 05:24:04.806138 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:04.806119    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c26b723-62af-41b3-bf3a-36a48065a924-istio-data\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm\" (UID: \"1c26b723-62af-41b3-bf3a-36a48065a924\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm"
Oct 30 05:24:04.806217 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:04.806194    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c26b723-62af-41b3-bf3a-36a48065a924-credential-socket\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm\" (UID: \"1c26b723-62af-41b3-bf3a-36a48065a924\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm"
Oct 30 05:24:04.806296 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:04.806278    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c26b723-62af-41b3-bf3a-36a48065a924-workload-socket\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm\" (UID: \"1c26b723-62af-41b3-bf3a-36a48065a924\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm"
Oct 30 05:24:04.806628 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:04.806604    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c26b723-62af-41b3-bf3a-36a48065a924-istiod-ca-cert\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm\" (UID: \"1c26b723-62af-41b3-bf3a-36a48065a924\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm"
Oct 30 05:24:04.807871 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:04.807845    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c26b723-62af-41b3-bf3a-36a48065a924-istio-envoy\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm\" (UID: \"1c26b723-62af-41b3-bf3a-36a48065a924\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm"
Oct 30 05:24:04.808122 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:04.808104    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c26b723-62af-41b3-bf3a-36a48065a924-istio-podinfo\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm\" (UID: \"1c26b723-62af-41b3-bf3a-36a48065a924\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm"
Oct 30 05:24:04.825357 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:04.825331    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c26b723-62af-41b3-bf3a-36a48065a924-istio-token\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm\" (UID: \"1c26b723-62af-41b3-bf3a-36a48065a924\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm"
Oct 30 05:24:04.825478 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:04.825435    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6rcj\" (UniqueName: \"kubernetes.io/projected/1c26b723-62af-41b3-bf3a-36a48065a924-kube-api-access-f6rcj\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm\" (UID: \"1c26b723-62af-41b3-bf3a-36a48065a924\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm"
Oct 30 05:24:04.868170 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:04.868140    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm"
Oct 30 05:24:05.192801 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:05.192772    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm"]
Oct 30 05:24:05.195528 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:24:05.195496    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c26b723_62af_41b3_bf3a_36a48065a924.slice/crio-e581fd588f825cdb4c70355a11d45c6ee6a59e973ab7c6294225e83b2654809e WatchSource:0}: Error finding container e581fd588f825cdb4c70355a11d45c6ee6a59e973ab7c6294225e83b2654809e: Status 404 returned error can't find the container with id e581fd588f825cdb4c70355a11d45c6ee6a59e973ab7c6294225e83b2654809e
Oct 30 05:24:05.198528 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:05.198492    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:24:05.198624 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:05.198564    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:24:05.198624 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:05.198595    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:24:05.794283 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:05.794240    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm" event={"ID":"1c26b723-62af-41b3-bf3a-36a48065a924","Type":"ContainerStarted","Data":"cf231194720d27a81df6e09d49ec938fcc546d22628fff97866e69bdd16498bc"}
Oct 30 05:24:05.794732 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:05.794300    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm" event={"ID":"1c26b723-62af-41b3-bf3a-36a48065a924","Type":"ContainerStarted","Data":"e581fd588f825cdb4c70355a11d45c6ee6a59e973ab7c6294225e83b2654809e"}
Oct 30 05:24:05.858623 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:05.858574    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm" podStartSLOduration=1.8585609490000001 podStartE2EDuration="1.858560949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:24:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:24:05.844521849 +0000 UTC m=+2375.068341771" watchObservedRunningTime="2026-10-30 05:24:05.858560949 +0000 UTC m=+2375.082380850"
Oct 30 05:24:05.869078 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:05.869049    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm"
Oct 30 05:24:06.008550 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:06.008514    5898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 05:24:06.008732 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:06.008571    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48" podUID="66f813c3-c7c9-452d-839d-4710108f03c4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 05:24:06.062522 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:06.062411    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:24:06.871027 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:06.870994    5898 patch_prober.go:29] interesting pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 05:24:06.871376 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:06.871053    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm" podUID="1c26b723-62af-41b3-bf3a-36a48065a924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 05:24:07.114240 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:07.114208    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48"
Oct 30 05:24:07.237342 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:07.237305    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/66f813c3-c7c9-452d-839d-4710108f03c4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66f813c3-c7c9-452d-839d-4710108f03c4-istio-podinfo\") pod \"66f813c3-c7c9-452d-839d-4710108f03c4\" (UID: \"66f813c3-c7c9-452d-839d-4710108f03c4\") "
Oct 30 05:24:07.237515 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:07.237393    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66f813c3-c7c9-452d-839d-4710108f03c4-istio-token\" (UniqueName: \"kubernetes.io/projected/66f813c3-c7c9-452d-839d-4710108f03c4-istio-token\") pod \"66f813c3-c7c9-452d-839d-4710108f03c4\" (UID: \"66f813c3-c7c9-452d-839d-4710108f03c4\") "
Oct 30 05:24:07.237603 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:07.237583    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66f813c3-c7c9-452d-839d-4710108f03c4-kube-api-access-4g55g\" (UniqueName: \"kubernetes.io/projected/66f813c3-c7c9-452d-839d-4710108f03c4-kube-api-access-4g55g\") pod \"66f813c3-c7c9-452d-839d-4710108f03c4\" (UID: \"66f813c3-c7c9-452d-839d-4710108f03c4\") "
Oct 30 05:24:07.240203 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:07.237696    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66f813c3-c7c9-452d-839d-4710108f03c4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66f813c3-c7c9-452d-839d-4710108f03c4-credential-socket\") pod \"66f813c3-c7c9-452d-839d-4710108f03c4\" (UID: \"66f813c3-c7c9-452d-839d-4710108f03c4\") "
Oct 30 05:24:07.240203 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:07.237784    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66f813c3-c7c9-452d-839d-4710108f03c4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66f813c3-c7c9-452d-839d-4710108f03c4-istio-data\") pod \"66f813c3-c7c9-452d-839d-4710108f03c4\" (UID: \"66f813c3-c7c9-452d-839d-4710108f03c4\") "
Oct 30 05:24:07.240203 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:07.237835    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66f813c3-c7c9-452d-839d-4710108f03c4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66f813c3-c7c9-452d-839d-4710108f03c4-workload-certs\") pod \"66f813c3-c7c9-452d-839d-4710108f03c4\" (UID: \"66f813c3-c7c9-452d-839d-4710108f03c4\") "
Oct 30 05:24:07.240203 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:07.237939    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66f813c3-c7c9-452d-839d-4710108f03c4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66f813c3-c7c9-452d-839d-4710108f03c4-workload-socket\") pod \"66f813c3-c7c9-452d-839d-4710108f03c4\" (UID: \"66f813c3-c7c9-452d-839d-4710108f03c4\") "
Oct 30 05:24:07.240203 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:07.237999    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66f813c3-c7c9-452d-839d-4710108f03c4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66f813c3-c7c9-452d-839d-4710108f03c4-istio-envoy\") pod \"66f813c3-c7c9-452d-839d-4710108f03c4\" (UID: \"66f813c3-c7c9-452d-839d-4710108f03c4\") "
Oct 30 05:24:07.240203 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:07.238010    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66f813c3-c7c9-452d-839d-4710108f03c4-credential-socket" pod "66f813c3-c7c9-452d-839d-4710108f03c4" (UID: "66f813c3-c7c9-452d-839d-4710108f03c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:24:07.240203 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:07.238046    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66f813c3-c7c9-452d-839d-4710108f03c4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66f813c3-c7c9-452d-839d-4710108f03c4-istiod-ca-cert\") pod \"66f813c3-c7c9-452d-839d-4710108f03c4\" (UID: \"66f813c3-c7c9-452d-839d-4710108f03c4\") "
Oct 30 05:24:07.240203 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:07.238140    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66f813c3-c7c9-452d-839d-4710108f03c4-istio-data" pod "66f813c3-c7c9-452d-839d-4710108f03c4" (UID: "66f813c3-c7c9-452d-839d-4710108f03c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:24:07.240203 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:07.238240    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66f813c3-c7c9-452d-839d-4710108f03c4-workload-socket" pod "66f813c3-c7c9-452d-839d-4710108f03c4" (UID: "66f813c3-c7c9-452d-839d-4710108f03c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:24:07.240203 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:07.238415    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66f813c3-c7c9-452d-839d-4710108f03c4-workload-certs" pod "66f813c3-c7c9-452d-839d-4710108f03c4" (UID: "66f813c3-c7c9-452d-839d-4710108f03c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:24:07.240203 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:07.238590    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66f813c3-c7c9-452d-839d-4710108f03c4-istiod-ca-cert" pod "66f813c3-c7c9-452d-839d-4710108f03c4" (UID: "66f813c3-c7c9-452d-839d-4710108f03c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:24:07.240203 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:07.238676    5898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66f813c3-c7c9-452d-839d-4710108f03c4-credential-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:24:07.240203 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:07.238699    5898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66f813c3-c7c9-452d-839d-4710108f03c4-istio-data\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:24:07.240203 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:07.238714    5898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66f813c3-c7c9-452d-839d-4710108f03c4-workload-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:24:07.240203 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:07.238727    5898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66f813c3-c7c9-452d-839d-4710108f03c4-workload-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:24:07.240203 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:07.240017    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66f813c3-c7c9-452d-839d-4710108f03c4-istio-podinfo" pod "66f813c3-c7c9-452d-839d-4710108f03c4" (UID: "66f813c3-c7c9-452d-839d-4710108f03c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:24:07.241511 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:07.241484    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66f813c3-c7c9-452d-839d-4710108f03c4-kube-api-access-4g55g" pod "66f813c3-c7c9-452d-839d-4710108f03c4" (UID: "66f813c3-c7c9-452d-839d-4710108f03c4"). InnerVolumeSpecName "kube-api-access-4g55g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:24:07.241588 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:07.241509    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66f813c3-c7c9-452d-839d-4710108f03c4-istio-envoy" pod "66f813c3-c7c9-452d-839d-4710108f03c4" (UID: "66f813c3-c7c9-452d-839d-4710108f03c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:24:07.241648 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:07.241579    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66f813c3-c7c9-452d-839d-4710108f03c4-istio-token" pod "66f813c3-c7c9-452d-839d-4710108f03c4" (UID: "66f813c3-c7c9-452d-839d-4710108f03c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:24:07.339778 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:07.339736    5898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66f813c3-c7c9-452d-839d-4710108f03c4-istio-envoy\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:24:07.339778 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:07.339770    5898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66f813c3-c7c9-452d-839d-4710108f03c4-istiod-ca-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:24:07.339778 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:07.339783    5898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66f813c3-c7c9-452d-839d-4710108f03c4-istio-podinfo\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:24:07.340050 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:07.339795    5898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66f813c3-c7c9-452d-839d-4710108f03c4-istio-token\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:24:07.340050 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:07.339806    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g55g\" (UniqueName: \"kubernetes.io/projected/66f813c3-c7c9-452d-839d-4710108f03c4-kube-api-access-4g55g\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:24:07.800438 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:07.800336    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:24:07.813153 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:07.813118    5898 generic.go:356] "Generic (PLEG): container finished" podID="66f813c3-c7c9-452d-839d-4710108f03c4" containerID="3f2bbd9c32545fbf42a40c68f01534d2b515e7194ce1c896e7d5e310c3949e34" exitCode=0
Oct 30 05:24:07.813325 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:07.813168    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48"
Oct 30 05:24:07.813325 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:07.813220    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48" event={"ID":"66f813c3-c7c9-452d-839d-4710108f03c4","Type":"ContainerDied","Data":"3f2bbd9c32545fbf42a40c68f01534d2b515e7194ce1c896e7d5e310c3949e34"}
Oct 30 05:24:07.813325 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:07.813272    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48" event={"ID":"66f813c3-c7c9-452d-839d-4710108f03c4","Type":"ContainerDied","Data":"f3c5054b96800f0c624b96cdc2529f585f505185e34e1d20f720fd44b48cb7c4"}
Oct 30 05:24:07.813325 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:07.813301    5898 scope.go:122] "RemoveContainer" containerID="3f2bbd9c32545fbf42a40c68f01534d2b515e7194ce1c896e7d5e310c3949e34"
Oct 30 05:24:07.824943 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:07.824892    5898 scope.go:122] "RemoveContainer" containerID="3f2bbd9c32545fbf42a40c68f01534d2b515e7194ce1c896e7d5e310c3949e34"
Oct 30 05:24:07.825275 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:24:07.825249    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f2bbd9c32545fbf42a40c68f01534d2b515e7194ce1c896e7d5e310c3949e34\": container with ID starting with 3f2bbd9c32545fbf42a40c68f01534d2b515e7194ce1c896e7d5e310c3949e34 not found: ID does not exist" containerID="3f2bbd9c32545fbf42a40c68f01534d2b515e7194ce1c896e7d5e310c3949e34"
Oct 30 05:24:07.825368 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:07.825288    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f2bbd9c32545fbf42a40c68f01534d2b515e7194ce1c896e7d5e310c3949e34"} err="failed to get container status \"3f2bbd9c32545fbf42a40c68f01534d2b515e7194ce1c896e7d5e310c3949e34\": rpc error: code = NotFound desc = could not find container \"3f2bbd9c32545fbf42a40c68f01534d2b515e7194ce1c896e7d5e310c3949e34\": container with ID starting with 3f2bbd9c32545fbf42a40c68f01534d2b515e7194ce1c896e7d5e310c3949e34 not found: ID does not exist"
Oct 30 05:24:07.877925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:07.869619    5898 patch_prober.go:29] interesting pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 05:24:07.877925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:07.869702    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm" podUID="1c26b723-62af-41b3-bf3a-36a48065a924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 05:24:07.912924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:07.911954    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48"]
Oct 30 05:24:07.919767 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:07.919733    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-k9r48"]
Oct 30 05:24:08.868525 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:08.868490    5898 patch_prober.go:29] interesting pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 05:24:08.868707 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:08.868541    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm" podUID="1c26b723-62af-41b3-bf3a-36a48065a924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 05:24:08.909986 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:08.909952    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66f813c3-c7c9-452d-839d-4710108f03c4" path="/var/lib/kubelet/pods/66f813c3-c7c9-452d-839d-4710108f03c4/volumes"
Oct 30 05:24:09.573852 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:09.573811    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:24:09.869198 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:09.869110    5898 patch_prober.go:29] interesting pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 05:24:09.869198 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:09.869169    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm" podUID="1c26b723-62af-41b3-bf3a-36a48065a924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 05:24:10.770982 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:10.770947    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:24:10.869303 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:10.869268    5898 patch_prober.go:29] interesting pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 05:24:10.869487 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:10.869328    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm" podUID="1c26b723-62af-41b3-bf3a-36a48065a924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 05:24:11.739804 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:11.739763    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:24:11.873473 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:11.873447    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm"
Oct 30 05:24:11.873880 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:11.873847    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm"
Oct 30 05:24:11.874588 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:11.874570    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm"
Oct 30 05:24:13.632791 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:13.632756    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:24:14.590584 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:14.590547    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:24:15.389577 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:15.389538    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:24:17.176178 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:17.176131    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:24:19.030063 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:19.030025    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:24:19.131848 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:19.131814    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:24:19.225001 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:19.224952    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:24:20.493693 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:20.493644    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:24:20.746980 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:20.744702    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:24:20.841933 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:20.841867    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:24:21.015152 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:21.015063    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:24:21.222796 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:21.222758    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:24:22.697697 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:22.697663    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:24:24.551263 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:24.551228    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:24:26.248959 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:26.248914    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:24:26.819202 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:24:26.819155    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:24:27.834205 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:27.834169    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:24:29.525414 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:29.525377    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:24:29.819940 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:29.819719    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:24:29.926461 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:29.924429    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:24:31.352612 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:31.352578    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:24:32.727682 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:32.727649    5898 scope.go:122] "RemoveContainer" containerID="f98c5af65814cbcfd8b764f127433a568ac3688d64af0090424eb077e4ca293e"
Oct 30 05:24:33.037193 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:33.037090    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:24:34.857964 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:34.857923    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:24:35.467370 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:35.467336    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:24:36.491840 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:36.491809    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:24:38.103253 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:38.103198    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:24:39.946627 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:39.946569    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:24:41.698669 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:41.698626    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:24:42.229645 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:42.229616    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:24:43.392167 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:43.392125    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:24:44.347780 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:44.347737    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:24:45.250865 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:45.250830    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:24:46.675859 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:46.675824    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-802b8a-6bdbc888b-7645w"]
Oct 30 05:24:46.676291 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:46.676040    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-802b8a-6bdbc888b-7645w" podUID="27c6e30a-940d-4fcf-b229-16a706b90709" containerName="maas-api" containerID="cri-o://318a9520eba0ed81aaca02474bd53b2a92fda138d1c4f132cdb0572f02606a40" gracePeriod=30
Oct 30 05:24:46.738606 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:46.738419    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:24:47.132922 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:47.130545    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-802b8a-6bdbc888b-7645w"
Oct 30 05:24:47.274042 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:47.274004    5898 generic.go:356] "Generic (PLEG): container finished" podID="27c6e30a-940d-4fcf-b229-16a706b90709" containerID="318a9520eba0ed81aaca02474bd53b2a92fda138d1c4f132cdb0572f02606a40" exitCode=0
Oct 30 05:24:47.274288 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:47.274263    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-802b8a-6bdbc888b-7645w" event={"ID":"27c6e30a-940d-4fcf-b229-16a706b90709","Type":"ContainerDied","Data":"318a9520eba0ed81aaca02474bd53b2a92fda138d1c4f132cdb0572f02606a40"}
Oct 30 05:24:47.274387 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:47.274360    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-802b8a-6bdbc888b-7645w" event={"ID":"27c6e30a-940d-4fcf-b229-16a706b90709","Type":"ContainerDied","Data":"c84396b1c25ca8b085828be4f0417199c1a8000e15cb9baeb0585b7a7c2ad635"}
Oct 30 05:24:47.274447 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:47.274410    5898 scope.go:122] "RemoveContainer" containerID="318a9520eba0ed81aaca02474bd53b2a92fda138d1c4f132cdb0572f02606a40"
Oct 30 05:24:47.274668 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:47.274236    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-802b8a-6bdbc888b-7645w"
Oct 30 05:24:47.293290 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:47.292684    5898 scope.go:122] "RemoveContainer" containerID="318a9520eba0ed81aaca02474bd53b2a92fda138d1c4f132cdb0572f02606a40"
Oct 30 05:24:47.293384 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:24:47.293295    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"318a9520eba0ed81aaca02474bd53b2a92fda138d1c4f132cdb0572f02606a40\": container with ID starting with 318a9520eba0ed81aaca02474bd53b2a92fda138d1c4f132cdb0572f02606a40 not found: ID does not exist" containerID="318a9520eba0ed81aaca02474bd53b2a92fda138d1c4f132cdb0572f02606a40"
Oct 30 05:24:47.293384 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:47.293327    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"318a9520eba0ed81aaca02474bd53b2a92fda138d1c4f132cdb0572f02606a40"} err="failed to get container status \"318a9520eba0ed81aaca02474bd53b2a92fda138d1c4f132cdb0572f02606a40\": rpc error: code = NotFound desc = could not find container \"318a9520eba0ed81aaca02474bd53b2a92fda138d1c4f132cdb0572f02606a40\": container with ID starting with 318a9520eba0ed81aaca02474bd53b2a92fda138d1c4f132cdb0572f02606a40 not found: ID does not exist"
Oct 30 05:24:47.296955 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:47.295108    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:24:47.323912 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:47.323863    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27c6e30a-940d-4fcf-b229-16a706b90709-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27c6e30a-940d-4fcf-b229-16a706b90709-maas-api-tls\") pod \"27c6e30a-940d-4fcf-b229-16a706b90709\" (UID: \"27c6e30a-940d-4fcf-b229-16a706b90709\") "
Oct 30 05:24:47.324074 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:47.323954    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27c6e30a-940d-4fcf-b229-16a706b90709-metrics-certs\" (UniqueName: \"kubernetes.io/secret/27c6e30a-940d-4fcf-b229-16a706b90709-metrics-certs\") pod \"27c6e30a-940d-4fcf-b229-16a706b90709\" (UID: \"27c6e30a-940d-4fcf-b229-16a706b90709\") "
Oct 30 05:24:47.324074 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:47.323988    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27c6e30a-940d-4fcf-b229-16a706b90709-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27c6e30a-940d-4fcf-b229-16a706b90709-openshift-service-ca\") pod \"27c6e30a-940d-4fcf-b229-16a706b90709\" (UID: \"27c6e30a-940d-4fcf-b229-16a706b90709\") "
Oct 30 05:24:47.324195 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:47.324153    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27c6e30a-940d-4fcf-b229-16a706b90709-kube-api-access-rgbrf\" (UniqueName: \"kubernetes.io/projected/27c6e30a-940d-4fcf-b229-16a706b90709-kube-api-access-rgbrf\") pod \"27c6e30a-940d-4fcf-b229-16a706b90709\" (UID: \"27c6e30a-940d-4fcf-b229-16a706b90709\") "
Oct 30 05:24:47.324411 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:47.324384    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27c6e30a-940d-4fcf-b229-16a706b90709-openshift-service-ca" pod "27c6e30a-940d-4fcf-b229-16a706b90709" (UID: "27c6e30a-940d-4fcf-b229-16a706b90709"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:24:47.324698 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:47.324678    5898 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27c6e30a-940d-4fcf-b229-16a706b90709-openshift-service-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:24:47.326086 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:47.326064    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27c6e30a-940d-4fcf-b229-16a706b90709-maas-api-tls" pod "27c6e30a-940d-4fcf-b229-16a706b90709" (UID: "27c6e30a-940d-4fcf-b229-16a706b90709"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:24:47.326255 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:47.326240    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27c6e30a-940d-4fcf-b229-16a706b90709-kube-api-access-rgbrf" pod "27c6e30a-940d-4fcf-b229-16a706b90709" (UID: "27c6e30a-940d-4fcf-b229-16a706b90709"). InnerVolumeSpecName "kube-api-access-rgbrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:24:47.326363 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:47.326349    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27c6e30a-940d-4fcf-b229-16a706b90709-metrics-certs" pod "27c6e30a-940d-4fcf-b229-16a706b90709" (UID: "27c6e30a-940d-4fcf-b229-16a706b90709"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:24:47.410126 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:47.410080    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:24:47.425508 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:47.425476    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgbrf\" (UniqueName: \"kubernetes.io/projected/27c6e30a-940d-4fcf-b229-16a706b90709-kube-api-access-rgbrf\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:24:47.425508 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:47.425507    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27c6e30a-940d-4fcf-b229-16a706b90709-maas-api-tls\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:24:47.425822 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:47.425522    5898 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27c6e30a-940d-4fcf-b229-16a706b90709-metrics-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:24:47.510213 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:47.510161    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:24:47.621970 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:47.621937    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-802b8a-6bdbc888b-7645w"]
Oct 30 05:24:47.625347 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:47.625318    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-802b8a-6bdbc888b-7645w"]
Oct 30 05:24:47.998702 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:47.998627    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:24:48.825808 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:48.825773    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:24:48.892464 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:48.892433    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27c6e30a-940d-4fcf-b229-16a706b90709" path="/var/lib/kubelet/pods/27c6e30a-940d-4fcf-b229-16a706b90709/volumes"
Oct 30 05:24:49.284676 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:49.284591    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:24:51.556669 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:51.556638    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:24:53.289113 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:53.289073    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:24:53.450677 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:53.450642    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:24:54.429101 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:54.429070    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:24:55.038166 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:55.038134    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:24:55.170642 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:55.170611    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:24:55.454160 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:55.454131    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:24:55.609967 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:55.609522    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:24:55.737607 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:55.737525    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:24:55.967764 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:55.967716    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:24:56.819536 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:24:56.819501    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:24:57.803815 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:57.803759    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:24:59.227650 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:59.227619    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:24:59.348512 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:59.348478    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:24:59.505383 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:24:59.505305    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:25:00.689300 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:00.689267    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:25:01.084521 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:01.084481    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:25:01.190102 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:01.190069    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:25:02.583134 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:02.583098    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:25:04.561469 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:04.561417    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:25:06.226822 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:06.226791    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:25:06.461219 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:06.461186    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:25:06.571381 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:06.571290    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:25:08.049153 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:08.049113    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:25:08.497480 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:08.497443    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:25:08.871522 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:08.871486    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5"]
Oct 30 05:25:08.872409 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:08.872382    5898 cpu_manager.go:414] "containerMap: removing container" podUID="66f813c3-c7c9-452d-839d-4710108f03c4" containerName="istio-proxy"
Oct 30 05:25:08.872517 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:08.872411    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66f813c3-c7c9-452d-839d-4710108f03c4" containerName="istio-proxy"
Oct 30 05:25:08.872517 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:08.872446    5898 cpu_manager.go:414] "containerMap: removing container" podUID="27c6e30a-940d-4fcf-b229-16a706b90709" containerName="maas-api"
Oct 30 05:25:08.872517 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:08.872454    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27c6e30a-940d-4fcf-b229-16a706b90709" containerName="maas-api"
Oct 30 05:25:08.872626 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:08.872597    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="27c6e30a-940d-4fcf-b229-16a706b90709" containerName="maas-api"
Oct 30 05:25:08.872626 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:08.872620    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="66f813c3-c7c9-452d-839d-4710108f03c4" containerName="istio-proxy"
Oct 30 05:25:08.877064 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:08.877023    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5"
Oct 30 05:25:08.899314 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:08.899282    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5"]
Oct 30 05:25:08.904948 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:08.904922    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-93bad9-openshift-default-dockercfg-hnpf6\""
Oct 30 05:25:08.937438 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:08.937404    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59cf2966-7359-48ba-bd83-769f0c076397-workload-socket\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5\" (UID: \"59cf2966-7359-48ba-bd83-769f0c076397\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5"
Oct 30 05:25:08.937611 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:08.937446    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59cf2966-7359-48ba-bd83-769f0c076397-workload-certs\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5\" (UID: \"59cf2966-7359-48ba-bd83-769f0c076397\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5"
Oct 30 05:25:08.937611 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:08.937479    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45nn7\" (UniqueName: \"kubernetes.io/projected/59cf2966-7359-48ba-bd83-769f0c076397-kube-api-access-45nn7\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5\" (UID: \"59cf2966-7359-48ba-bd83-769f0c076397\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5"
Oct 30 05:25:08.937780 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:08.937613    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59cf2966-7359-48ba-bd83-769f0c076397-credential-socket\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5\" (UID: \"59cf2966-7359-48ba-bd83-769f0c076397\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5"
Oct 30 05:25:08.937780 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:08.937711    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59cf2966-7359-48ba-bd83-769f0c076397-istio-data\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5\" (UID: \"59cf2966-7359-48ba-bd83-769f0c076397\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5"
Oct 30 05:25:08.937959 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:08.937916    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59cf2966-7359-48ba-bd83-769f0c076397-istio-envoy\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5\" (UID: \"59cf2966-7359-48ba-bd83-769f0c076397\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5"
Oct 30 05:25:08.938038 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:08.937970    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59cf2966-7359-48ba-bd83-769f0c076397-istio-token\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5\" (UID: \"59cf2966-7359-48ba-bd83-769f0c076397\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5"
Oct 30 05:25:08.938135 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:08.938115    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59cf2966-7359-48ba-bd83-769f0c076397-istio-podinfo\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5\" (UID: \"59cf2966-7359-48ba-bd83-769f0c076397\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5"
Oct 30 05:25:08.938264 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:08.938244    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59cf2966-7359-48ba-bd83-769f0c076397-istiod-ca-cert\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5\" (UID: \"59cf2966-7359-48ba-bd83-769f0c076397\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5"
Oct 30 05:25:09.039343 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:09.039309    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59cf2966-7359-48ba-bd83-769f0c076397-istio-podinfo\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5\" (UID: \"59cf2966-7359-48ba-bd83-769f0c076397\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5"
Oct 30 05:25:09.039510 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:09.039385    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59cf2966-7359-48ba-bd83-769f0c076397-istiod-ca-cert\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5\" (UID: \"59cf2966-7359-48ba-bd83-769f0c076397\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5"
Oct 30 05:25:09.039510 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:09.039498    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59cf2966-7359-48ba-bd83-769f0c076397-workload-socket\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5\" (UID: \"59cf2966-7359-48ba-bd83-769f0c076397\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5"
Oct 30 05:25:09.039672 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:09.039521    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59cf2966-7359-48ba-bd83-769f0c076397-workload-certs\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5\" (UID: \"59cf2966-7359-48ba-bd83-769f0c076397\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5"
Oct 30 05:25:09.039672 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:09.039549    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45nn7\" (UniqueName: \"kubernetes.io/projected/59cf2966-7359-48ba-bd83-769f0c076397-kube-api-access-45nn7\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5\" (UID: \"59cf2966-7359-48ba-bd83-769f0c076397\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5"
Oct 30 05:25:09.039672 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:09.039598    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59cf2966-7359-48ba-bd83-769f0c076397-credential-socket\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5\" (UID: \"59cf2966-7359-48ba-bd83-769f0c076397\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5"
Oct 30 05:25:09.039672 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:09.039661    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59cf2966-7359-48ba-bd83-769f0c076397-istio-data\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5\" (UID: \"59cf2966-7359-48ba-bd83-769f0c076397\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5"
Oct 30 05:25:09.039883 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:09.039710    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59cf2966-7359-48ba-bd83-769f0c076397-istio-envoy\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5\" (UID: \"59cf2966-7359-48ba-bd83-769f0c076397\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5"
Oct 30 05:25:09.039883 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:09.039745    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59cf2966-7359-48ba-bd83-769f0c076397-istio-token\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5\" (UID: \"59cf2966-7359-48ba-bd83-769f0c076397\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5"
Oct 30 05:25:09.040022 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:09.039957    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59cf2966-7359-48ba-bd83-769f0c076397-workload-certs\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5\" (UID: \"59cf2966-7359-48ba-bd83-769f0c076397\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5"
Oct 30 05:25:09.040022 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:09.039972    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59cf2966-7359-48ba-bd83-769f0c076397-workload-socket\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5\" (UID: \"59cf2966-7359-48ba-bd83-769f0c076397\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5"
Oct 30 05:25:09.040201 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:09.040177    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59cf2966-7359-48ba-bd83-769f0c076397-istio-data\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5\" (UID: \"59cf2966-7359-48ba-bd83-769f0c076397\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5"
Oct 30 05:25:09.040402 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:09.040387    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59cf2966-7359-48ba-bd83-769f0c076397-istiod-ca-cert\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5\" (UID: \"59cf2966-7359-48ba-bd83-769f0c076397\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5"
Oct 30 05:25:09.040512 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:09.040489    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59cf2966-7359-48ba-bd83-769f0c076397-credential-socket\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5\" (UID: \"59cf2966-7359-48ba-bd83-769f0c076397\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5"
Oct 30 05:25:09.042245 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:09.042125    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59cf2966-7359-48ba-bd83-769f0c076397-istio-envoy\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5\" (UID: \"59cf2966-7359-48ba-bd83-769f0c076397\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5"
Oct 30 05:25:09.042395 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:09.042369    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59cf2966-7359-48ba-bd83-769f0c076397-istio-podinfo\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5\" (UID: \"59cf2966-7359-48ba-bd83-769f0c076397\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5"
Oct 30 05:25:09.062134 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:09.062111    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59cf2966-7359-48ba-bd83-769f0c076397-istio-token\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5\" (UID: \"59cf2966-7359-48ba-bd83-769f0c076397\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5"
Oct 30 05:25:09.062449 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:09.062405    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45nn7\" (UniqueName: \"kubernetes.io/projected/59cf2966-7359-48ba-bd83-769f0c076397-kube-api-access-45nn7\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5\" (UID: \"59cf2966-7359-48ba-bd83-769f0c076397\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5"
Oct 30 05:25:09.193960 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:09.193882    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5"
Oct 30 05:25:09.518158 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:09.518084    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5"]
Oct 30 05:25:09.521655 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:25:09.521623    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59cf2966_7359_48ba_bd83_769f0c076397.slice/crio-f18d6510e05120b9121fd0842ba7bdf47ef99bb2d3803e3ea92c49e737235e22 WatchSource:0}: Error finding container f18d6510e05120b9121fd0842ba7bdf47ef99bb2d3803e3ea92c49e737235e22: Status 404 returned error can't find the container with id f18d6510e05120b9121fd0842ba7bdf47ef99bb2d3803e3ea92c49e737235e22
Oct 30 05:25:09.524431 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:09.524394    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:25:09.524552 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:09.524465    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:25:09.524552 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:09.524497    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:25:10.065264 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:10.065221    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:25:10.415046 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:10.415013    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h"]
Oct 30 05:25:10.419355 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:10.419338    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h"
Oct 30 05:25:10.446122 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:10.446092    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-e15c99-openshift-default-dockercfg-ngf5s\""
Oct 30 05:25:10.451325 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:10.451294    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h"]
Oct 30 05:25:10.527049 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:10.527009    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5" event={"ID":"59cf2966-7359-48ba-bd83-769f0c076397","Type":"ContainerStarted","Data":"db826a7fc82a89847819d280211b43bac38451cf91ea593a52b6faaac97fa63d"}
Oct 30 05:25:10.527225 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:10.527070    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5" event={"ID":"59cf2966-7359-48ba-bd83-769f0c076397","Type":"ContainerStarted","Data":"f18d6510e05120b9121fd0842ba7bdf47ef99bb2d3803e3ea92c49e737235e22"}
Oct 30 05:25:10.562324 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:10.562286    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c2a73f8-80fb-46e4-ac53-50ca8ed027d9-credential-socket\") pod \"e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h\" (UID: \"6c2a73f8-80fb-46e4-ac53-50ca8ed027d9\") " pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h"
Oct 30 05:25:10.562473 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:10.562347    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c2a73f8-80fb-46e4-ac53-50ca8ed027d9-istio-envoy\") pod \"e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h\" (UID: \"6c2a73f8-80fb-46e4-ac53-50ca8ed027d9\") " pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h"
Oct 30 05:25:10.562825 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:10.562803    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c2a73f8-80fb-46e4-ac53-50ca8ed027d9-istio-data\") pod \"e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h\" (UID: \"6c2a73f8-80fb-46e4-ac53-50ca8ed027d9\") " pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h"
Oct 30 05:25:10.562882 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:10.562834    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c2a73f8-80fb-46e4-ac53-50ca8ed027d9-istio-token\") pod \"e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h\" (UID: \"6c2a73f8-80fb-46e4-ac53-50ca8ed027d9\") " pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h"
Oct 30 05:25:10.562882 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:10.562859    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rph6m\" (UniqueName: \"kubernetes.io/projected/6c2a73f8-80fb-46e4-ac53-50ca8ed027d9-kube-api-access-rph6m\") pod \"e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h\" (UID: \"6c2a73f8-80fb-46e4-ac53-50ca8ed027d9\") " pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h"
Oct 30 05:25:10.563044 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:10.563025    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c2a73f8-80fb-46e4-ac53-50ca8ed027d9-istiod-ca-cert\") pod \"e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h\" (UID: \"6c2a73f8-80fb-46e4-ac53-50ca8ed027d9\") " pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h"
Oct 30 05:25:10.563097 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:10.563086    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c2a73f8-80fb-46e4-ac53-50ca8ed027d9-workload-socket\") pod \"e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h\" (UID: \"6c2a73f8-80fb-46e4-ac53-50ca8ed027d9\") " pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h"
Oct 30 05:25:10.563141 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:10.563119    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c2a73f8-80fb-46e4-ac53-50ca8ed027d9-istio-podinfo\") pod \"e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h\" (UID: \"6c2a73f8-80fb-46e4-ac53-50ca8ed027d9\") " pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h"
Oct 30 05:25:10.563187 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:10.563139    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c2a73f8-80fb-46e4-ac53-50ca8ed027d9-workload-certs\") pod \"e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h\" (UID: \"6c2a73f8-80fb-46e4-ac53-50ca8ed027d9\") " pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h"
Oct 30 05:25:10.586929 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:10.574477    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5" podStartSLOduration=2.574461072 podStartE2EDuration="2.574461072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:25:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:25:10.570976406 +0000 UTC m=+2439.794796327" watchObservedRunningTime="2026-10-30 05:25:10.574461072 +0000 UTC m=+2439.798280976"
Oct 30 05:25:10.664020 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:10.663978    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c2a73f8-80fb-46e4-ac53-50ca8ed027d9-istio-data\") pod \"e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h\" (UID: \"6c2a73f8-80fb-46e4-ac53-50ca8ed027d9\") " pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h"
Oct 30 05:25:10.664020 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:10.664021    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c2a73f8-80fb-46e4-ac53-50ca8ed027d9-istio-token\") pod \"e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h\" (UID: \"6c2a73f8-80fb-46e4-ac53-50ca8ed027d9\") " pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h"
Oct 30 05:25:10.664265 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:10.664079    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rph6m\" (UniqueName: \"kubernetes.io/projected/6c2a73f8-80fb-46e4-ac53-50ca8ed027d9-kube-api-access-rph6m\") pod \"e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h\" (UID: \"6c2a73f8-80fb-46e4-ac53-50ca8ed027d9\") " pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h"
Oct 30 05:25:10.664265 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:10.664172    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c2a73f8-80fb-46e4-ac53-50ca8ed027d9-istiod-ca-cert\") pod \"e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h\" (UID: \"6c2a73f8-80fb-46e4-ac53-50ca8ed027d9\") " pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h"
Oct 30 05:25:10.664265 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:10.664212    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c2a73f8-80fb-46e4-ac53-50ca8ed027d9-workload-socket\") pod \"e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h\" (UID: \"6c2a73f8-80fb-46e4-ac53-50ca8ed027d9\") " pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h"
Oct 30 05:25:10.664265 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:10.664251    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c2a73f8-80fb-46e4-ac53-50ca8ed027d9-istio-podinfo\") pod \"e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h\" (UID: \"6c2a73f8-80fb-46e4-ac53-50ca8ed027d9\") " pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h"
Oct 30 05:25:10.664483 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:10.664268    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c2a73f8-80fb-46e4-ac53-50ca8ed027d9-workload-certs\") pod \"e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h\" (UID: \"6c2a73f8-80fb-46e4-ac53-50ca8ed027d9\") " pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h"
Oct 30 05:25:10.664483 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:10.664327    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c2a73f8-80fb-46e4-ac53-50ca8ed027d9-credential-socket\") pod \"e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h\" (UID: \"6c2a73f8-80fb-46e4-ac53-50ca8ed027d9\") " pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h"
Oct 30 05:25:10.664483 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:10.664367    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c2a73f8-80fb-46e4-ac53-50ca8ed027d9-istio-envoy\") pod \"e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h\" (UID: \"6c2a73f8-80fb-46e4-ac53-50ca8ed027d9\") " pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h"
Oct 30 05:25:10.664657 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:10.664480    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c2a73f8-80fb-46e4-ac53-50ca8ed027d9-istio-data\") pod \"e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h\" (UID: \"6c2a73f8-80fb-46e4-ac53-50ca8ed027d9\") " pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h"
Oct 30 05:25:10.664789 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:10.664764    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c2a73f8-80fb-46e4-ac53-50ca8ed027d9-workload-certs\") pod \"e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h\" (UID: \"6c2a73f8-80fb-46e4-ac53-50ca8ed027d9\") " pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h"
Oct 30 05:25:10.665131 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:10.665065    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c2a73f8-80fb-46e4-ac53-50ca8ed027d9-workload-socket\") pod \"e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h\" (UID: \"6c2a73f8-80fb-46e4-ac53-50ca8ed027d9\") " pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h"
Oct 30 05:25:10.665131 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:10.665100    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c2a73f8-80fb-46e4-ac53-50ca8ed027d9-credential-socket\") pod \"e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h\" (UID: \"6c2a73f8-80fb-46e4-ac53-50ca8ed027d9\") " pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h"
Oct 30 05:25:10.665275 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:10.665139    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c2a73f8-80fb-46e4-ac53-50ca8ed027d9-istiod-ca-cert\") pod \"e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h\" (UID: \"6c2a73f8-80fb-46e4-ac53-50ca8ed027d9\") " pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h"
Oct 30 05:25:10.666967 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:10.666942    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c2a73f8-80fb-46e4-ac53-50ca8ed027d9-istio-envoy\") pod \"e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h\" (UID: \"6c2a73f8-80fb-46e4-ac53-50ca8ed027d9\") " pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h"
Oct 30 05:25:10.667173 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:10.667156    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c2a73f8-80fb-46e4-ac53-50ca8ed027d9-istio-podinfo\") pod \"e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h\" (UID: \"6c2a73f8-80fb-46e4-ac53-50ca8ed027d9\") " pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h"
Oct 30 05:25:10.702795 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:10.702758    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rph6m\" (UniqueName: \"kubernetes.io/projected/6c2a73f8-80fb-46e4-ac53-50ca8ed027d9-kube-api-access-rph6m\") pod \"e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h\" (UID: \"6c2a73f8-80fb-46e4-ac53-50ca8ed027d9\") " pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h"
Oct 30 05:25:10.704728 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:10.704700    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c2a73f8-80fb-46e4-ac53-50ca8ed027d9-istio-token\") pod \"e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h\" (UID: \"6c2a73f8-80fb-46e4-ac53-50ca8ed027d9\") " pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h"
Oct 30 05:25:10.736784 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:10.736752    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h"
Oct 30 05:25:11.101504 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:11.101470    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h"]
Oct 30 05:25:11.105390 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:25:11.105360    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c2a73f8_80fb_46e4_ac53_50ca8ed027d9.slice/crio-53aac1f3b976d8cf13795b55602adca9c8edbc71009da5420ce31433332ae0c4 WatchSource:0}: Error finding container 53aac1f3b976d8cf13795b55602adca9c8edbc71009da5420ce31433332ae0c4: Status 404 returned error can't find the container with id 53aac1f3b976d8cf13795b55602adca9c8edbc71009da5420ce31433332ae0c4
Oct 30 05:25:11.108262 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:11.108231    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:25:11.108351 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:11.108296    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:25:11.108351 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:11.108335    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:25:11.198946 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:11.195383    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5"
Oct 30 05:25:11.199219 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:11.199026    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Oct 30 05:25:11.199219 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:11.199091    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5" podUID="59cf2966-7359-48ba-bd83-769f0c076397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Oct 30 05:25:11.535955 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:11.535856    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h" event={"ID":"6c2a73f8-80fb-46e4-ac53-50ca8ed027d9","Type":"ContainerStarted","Data":"80ffd4c0bfdb6404bbba58983a4fe831f9ca4e0c7a4e4880c200434081160f34"}
Oct 30 05:25:11.535955 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:11.535931    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h" event={"ID":"6c2a73f8-80fb-46e4-ac53-50ca8ed027d9","Type":"ContainerStarted","Data":"53aac1f3b976d8cf13795b55602adca9c8edbc71009da5420ce31433332ae0c4"}
Oct 30 05:25:11.596925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:11.594949    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h" podStartSLOduration=1.5949287 podStartE2EDuration="1.5949287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:25:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:25:11.5933984 +0000 UTC m=+2440.817218320" watchObservedRunningTime="2026-10-30 05:25:11.5949287 +0000 UTC m=+2440.818748604"
Oct 30 05:25:11.737324 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:11.737293    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h"
Oct 30 05:25:11.787979 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:11.787394    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:25:12.211505 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:12.211024    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Oct 30 05:25:12.211505 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:12.211092    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5" podUID="59cf2966-7359-48ba-bd83-769f0c076397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Oct 30 05:25:12.738544 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:12.738510    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Oct 30 05:25:12.738723 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:12.738563    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h" podUID="6c2a73f8-80fb-46e4-ac53-50ca8ed027d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Oct 30 05:25:13.194923 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:13.194340    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Oct 30 05:25:13.194923 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:13.194404    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5" podUID="59cf2966-7359-48ba-bd83-769f0c076397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Oct 30 05:25:13.532826 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:13.532742    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:25:13.737439 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:13.737401    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Oct 30 05:25:13.737611 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:13.737454    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h" podUID="6c2a73f8-80fb-46e4-ac53-50ca8ed027d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Oct 30 05:25:14.227219 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:14.227182    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Oct 30 05:25:14.227625 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:14.227594    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5" podUID="59cf2966-7359-48ba-bd83-769f0c076397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Oct 30 05:25:14.737489 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:14.737443    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Oct 30 05:25:14.737887 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:14.737508    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h" podUID="6c2a73f8-80fb-46e4-ac53-50ca8ed027d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Oct 30 05:25:15.196275 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:15.196239    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Oct 30 05:25:15.196589 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:15.196540    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5" podUID="59cf2966-7359-48ba-bd83-769f0c076397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Oct 30 05:25:15.351531 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:15.351496    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:25:15.737429 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:15.737390    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Oct 30 05:25:15.737624 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:15.737456    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h" podUID="6c2a73f8-80fb-46e4-ac53-50ca8ed027d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Oct 30 05:25:16.204051 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:16.204016    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Oct 30 05:25:16.204260 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:16.204089    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5" podUID="59cf2966-7359-48ba-bd83-769f0c076397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Oct 30 05:25:16.739718 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:16.739680    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Oct 30 05:25:16.740148 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:16.739732    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h" podUID="6c2a73f8-80fb-46e4-ac53-50ca8ed027d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Oct 30 05:25:17.202957 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:17.202926    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5"
Oct 30 05:25:17.203326 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:17.203303    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5"
Oct 30 05:25:17.203997 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:17.203977    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5"
Oct 30 05:25:17.296465 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:17.296431    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:25:17.738105 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:17.738070    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Oct 30 05:25:17.738285 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:17.738124    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h" podUID="6c2a73f8-80fb-46e4-ac53-50ca8ed027d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Oct 30 05:25:18.280485 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:18.280451    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:25:18.744169 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:18.744134    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h"
Oct 30 05:25:18.744999 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:18.744883    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h"
Oct 30 05:25:18.745768 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:18.745749    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h"
Oct 30 05:25:18.787046 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:18.787016    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:25:18.985924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:18.959538    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:25:19.295718 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:19.295666    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:25:21.209798 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:21.209753    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:25:22.828557 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:22.828520    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:25:23.017231 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:23.017195    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:25:23.126492 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:23.126401    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:25:23.232309 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:23.232278    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:25:24.685934 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:24.685881    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:25:24.784439 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:24.784405    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:25:25.259648 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:25.259619    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:25:26.818432 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:25:26.818394    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:25:27.043412 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:27.043375    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:25:28.276065 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:28.276034    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb"]
Oct 30 05:25:28.280279 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:28.280257    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb"
Oct 30 05:25:28.299364 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:28.299343    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb"]
Oct 30 05:25:28.307323 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:28.307297    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-ff4760-openshift-default-dockercfg-5kn59\""
Oct 30 05:25:28.379492 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:28.379457    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d77a7956-ec67-4cfb-ba98-fd1e6ea6f227-workload-socket\") pod \"e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb\" (UID: \"d77a7956-ec67-4cfb-ba98-fd1e6ea6f227\") " pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb"
Oct 30 05:25:28.379694 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:28.379505    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d77a7956-ec67-4cfb-ba98-fd1e6ea6f227-istio-token\") pod \"e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb\" (UID: \"d77a7956-ec67-4cfb-ba98-fd1e6ea6f227\") " pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb"
Oct 30 05:25:28.379694 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:28.379525    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d77a7956-ec67-4cfb-ba98-fd1e6ea6f227-istiod-ca-cert\") pod \"e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb\" (UID: \"d77a7956-ec67-4cfb-ba98-fd1e6ea6f227\") " pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb"
Oct 30 05:25:28.379816 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:28.379694    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d77a7956-ec67-4cfb-ba98-fd1e6ea6f227-istio-data\") pod \"e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb\" (UID: \"d77a7956-ec67-4cfb-ba98-fd1e6ea6f227\") " pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb"
Oct 30 05:25:28.379816 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:28.379796    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d77a7956-ec67-4cfb-ba98-fd1e6ea6f227-istio-podinfo\") pod \"e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb\" (UID: \"d77a7956-ec67-4cfb-ba98-fd1e6ea6f227\") " pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb"
Oct 30 05:25:28.379895 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:28.379850    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tgp7\" (UniqueName: \"kubernetes.io/projected/d77a7956-ec67-4cfb-ba98-fd1e6ea6f227-kube-api-access-8tgp7\") pod \"e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb\" (UID: \"d77a7956-ec67-4cfb-ba98-fd1e6ea6f227\") " pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb"
Oct 30 05:25:28.379895 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:28.379879    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d77a7956-ec67-4cfb-ba98-fd1e6ea6f227-istio-envoy\") pod \"e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb\" (UID: \"d77a7956-ec67-4cfb-ba98-fd1e6ea6f227\") " pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb"
Oct 30 05:25:28.380036 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:28.380015    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d77a7956-ec67-4cfb-ba98-fd1e6ea6f227-credential-socket\") pod \"e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb\" (UID: \"d77a7956-ec67-4cfb-ba98-fd1e6ea6f227\") " pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb"
Oct 30 05:25:28.380140 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:28.380126    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d77a7956-ec67-4cfb-ba98-fd1e6ea6f227-workload-certs\") pod \"e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb\" (UID: \"d77a7956-ec67-4cfb-ba98-fd1e6ea6f227\") " pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb"
Oct 30 05:25:28.481634 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:28.481602    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d77a7956-ec67-4cfb-ba98-fd1e6ea6f227-workload-certs\") pod \"e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb\" (UID: \"d77a7956-ec67-4cfb-ba98-fd1e6ea6f227\") " pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb"
Oct 30 05:25:28.481814 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:28.481692    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d77a7956-ec67-4cfb-ba98-fd1e6ea6f227-workload-socket\") pod \"e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb\" (UID: \"d77a7956-ec67-4cfb-ba98-fd1e6ea6f227\") " pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb"
Oct 30 05:25:28.481814 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:28.481736    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d77a7956-ec67-4cfb-ba98-fd1e6ea6f227-istio-token\") pod \"e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb\" (UID: \"d77a7956-ec67-4cfb-ba98-fd1e6ea6f227\") " pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb"
Oct 30 05:25:28.481814 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:28.481758    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d77a7956-ec67-4cfb-ba98-fd1e6ea6f227-istiod-ca-cert\") pod \"e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb\" (UID: \"d77a7956-ec67-4cfb-ba98-fd1e6ea6f227\") " pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb"
Oct 30 05:25:28.482031 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:28.481818    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d77a7956-ec67-4cfb-ba98-fd1e6ea6f227-istio-data\") pod \"e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb\" (UID: \"d77a7956-ec67-4cfb-ba98-fd1e6ea6f227\") " pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb"
Oct 30 05:25:28.482031 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:28.481888    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d77a7956-ec67-4cfb-ba98-fd1e6ea6f227-istio-podinfo\") pod \"e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb\" (UID: \"d77a7956-ec67-4cfb-ba98-fd1e6ea6f227\") " pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb"
Oct 30 05:25:28.482031 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:28.481966    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tgp7\" (UniqueName: \"kubernetes.io/projected/d77a7956-ec67-4cfb-ba98-fd1e6ea6f227-kube-api-access-8tgp7\") pod \"e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb\" (UID: \"d77a7956-ec67-4cfb-ba98-fd1e6ea6f227\") " pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb"
Oct 30 05:25:28.482031 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:28.482008    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d77a7956-ec67-4cfb-ba98-fd1e6ea6f227-istio-envoy\") pod \"e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb\" (UID: \"d77a7956-ec67-4cfb-ba98-fd1e6ea6f227\") " pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb"
Oct 30 05:25:28.482167 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:28.482069    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d77a7956-ec67-4cfb-ba98-fd1e6ea6f227-credential-socket\") pod \"e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb\" (UID: \"d77a7956-ec67-4cfb-ba98-fd1e6ea6f227\") " pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb"
Oct 30 05:25:28.482387 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:28.482363    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d77a7956-ec67-4cfb-ba98-fd1e6ea6f227-workload-certs\") pod \"e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb\" (UID: \"d77a7956-ec67-4cfb-ba98-fd1e6ea6f227\") " pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb"
Oct 30 05:25:28.482570 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:28.482433    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d77a7956-ec67-4cfb-ba98-fd1e6ea6f227-workload-socket\") pod \"e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb\" (UID: \"d77a7956-ec67-4cfb-ba98-fd1e6ea6f227\") " pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb"
Oct 30 05:25:28.482570 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:28.482484    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d77a7956-ec67-4cfb-ba98-fd1e6ea6f227-istio-data\") pod \"e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb\" (UID: \"d77a7956-ec67-4cfb-ba98-fd1e6ea6f227\") " pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb"
Oct 30 05:25:28.482709 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:28.482601    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d77a7956-ec67-4cfb-ba98-fd1e6ea6f227-credential-socket\") pod \"e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb\" (UID: \"d77a7956-ec67-4cfb-ba98-fd1e6ea6f227\") " pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb"
Oct 30 05:25:28.482820 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:28.482793    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d77a7956-ec67-4cfb-ba98-fd1e6ea6f227-istiod-ca-cert\") pod \"e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb\" (UID: \"d77a7956-ec67-4cfb-ba98-fd1e6ea6f227\") " pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb"
Oct 30 05:25:28.484543 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:28.484516    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d77a7956-ec67-4cfb-ba98-fd1e6ea6f227-istio-envoy\") pod \"e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb\" (UID: \"d77a7956-ec67-4cfb-ba98-fd1e6ea6f227\") " pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb"
Oct 30 05:25:28.484706 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:28.484690    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d77a7956-ec67-4cfb-ba98-fd1e6ea6f227-istio-podinfo\") pod \"e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb\" (UID: \"d77a7956-ec67-4cfb-ba98-fd1e6ea6f227\") " pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb"
Oct 30 05:25:28.500712 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:28.500683    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tgp7\" (UniqueName: \"kubernetes.io/projected/d77a7956-ec67-4cfb-ba98-fd1e6ea6f227-kube-api-access-8tgp7\") pod \"e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb\" (UID: \"d77a7956-ec67-4cfb-ba98-fd1e6ea6f227\") " pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb"
Oct 30 05:25:28.500969 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:28.500953    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d77a7956-ec67-4cfb-ba98-fd1e6ea6f227-istio-token\") pod \"e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb\" (UID: \"d77a7956-ec67-4cfb-ba98-fd1e6ea6f227\") " pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb"
Oct 30 05:25:28.594360 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:28.594328    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb"
Oct 30 05:25:28.918468 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:28.918444    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb"]
Oct 30 05:25:28.921035 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:25:28.921008    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd77a7956_ec67_4cfb_ba98_fd1e6ea6f227.slice/crio-c5ea5f3d9bab3342ec250942960113cdb40f275995c90db3c7bd6591456c3af9 WatchSource:0}: Error finding container c5ea5f3d9bab3342ec250942960113cdb40f275995c90db3c7bd6591456c3af9: Status 404 returned error can't find the container with id c5ea5f3d9bab3342ec250942960113cdb40f275995c90db3c7bd6591456c3af9
Oct 30 05:25:28.924441 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:28.924404    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:25:28.924501 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:28.924485    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:25:28.924543 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:28.924529    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:25:29.010746 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:29.010715    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:25:29.678620 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:29.678582    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb" event={"ID":"d77a7956-ec67-4cfb-ba98-fd1e6ea6f227","Type":"ContainerStarted","Data":"109ca16652c0ca0ffb819d91c285f062ab1096be49790b362fc1ce97b1ebd68c"}
Oct 30 05:25:29.679037 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:29.678642    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb" event={"ID":"d77a7956-ec67-4cfb-ba98-fd1e6ea6f227","Type":"ContainerStarted","Data":"c5ea5f3d9bab3342ec250942960113cdb40f275995c90db3c7bd6591456c3af9"}
Oct 30 05:25:29.741970 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:29.741890    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb" podStartSLOduration=1.741876717 podStartE2EDuration="1.741876717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:25:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:25:29.729418038 +0000 UTC m=+2458.953237938" watchObservedRunningTime="2026-10-30 05:25:29.741876717 +0000 UTC m=+2458.965696617"
Oct 30 05:25:30.596627 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:30.596585    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb"
Oct 30 05:25:30.597019 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:30.596785    5898 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Oct 30 05:25:30.597217 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:30.597193    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb" podUID="d77a7956-ec67-4cfb-ba98-fd1e6ea6f227" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Oct 30 05:25:30.739822 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:30.739779    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:25:31.595282 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:31.595246    5898 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Oct 30 05:25:31.595586 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:31.595544    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb" podUID="d77a7956-ec67-4cfb-ba98-fd1e6ea6f227" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Oct 30 05:25:32.236821 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:32.236775    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:25:32.507851 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:32.507694    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:25:32.602403 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:32.602355    5898 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Oct 30 05:25:32.602688 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:32.602658    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb" podUID="d77a7956-ec67-4cfb-ba98-fd1e6ea6f227" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Oct 30 05:25:33.599532 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:33.599344    5898 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Oct 30 05:25:33.599532 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:33.599400    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb" podUID="d77a7956-ec67-4cfb-ba98-fd1e6ea6f227" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Oct 30 05:25:34.597925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:34.597298    5898 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Oct 30 05:25:34.597925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:34.597362    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb" podUID="d77a7956-ec67-4cfb-ba98-fd1e6ea6f227" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Oct 30 05:25:34.805326 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:34.805291    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:25:35.594876 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:35.594844    5898 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Oct 30 05:25:35.595080 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:35.594933    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb" podUID="d77a7956-ec67-4cfb-ba98-fd1e6ea6f227" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Oct 30 05:25:36.600837 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:36.600806    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb"
Oct 30 05:25:36.601262 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:36.601238    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb"
Oct 30 05:25:36.602010 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:36.601991    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb"
Oct 30 05:25:36.794276 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:36.794242    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:25:38.698304 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:38.698273    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:25:40.607408 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:40.607375    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:25:41.189297 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:41.189263    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:25:42.346925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:42.344104    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:25:43.055258 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:43.055218    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:25:43.152798 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:43.152754    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:25:45.006154 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:45.006119    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:25:46.636240 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:46.636198    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:25:47.340116 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:47.340083    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:25:47.452225 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:47.452191    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:25:47.573242 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:47.573205    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:25:47.989059 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:47.989024    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:25:49.297137 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:49.297100    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:25:51.103453 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:51.103420    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:25:53.078612 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:53.078582    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:25:54.715708 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:54.715664    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:25:54.832981 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:54.832940    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:25:54.940943 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:54.940891    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:25:55.117421 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:55.117388    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242"]
Oct 30 05:25:55.117935 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:55.117886    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242" podUID="5df41ffb-123b-45fd-a3ae-3c61bcaafd5a" containerName="istio-proxy" containerID="cri-o://8432784f5544c48a0c29699cdeecfd6a240a51c9f80e367d5bb3995bc48bde10" gracePeriod=30
Oct 30 05:25:55.124992 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:55.124951    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:25:55.899227 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:55.899189    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:25:56.443522 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:56.443486    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:25:56.580926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:56.577505    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:25:56.686429 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:56.686394    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:25:56.835273 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:25:56.835241    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:25:58.655749 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:25:58.655712    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:26:00.232037 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:00.231769    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:26:00.316411 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:00.316385    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242"
Oct 30 05:26:00.460416 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:00.460331    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5df41ffb-123b-45fd-a3ae-3c61bcaafd5a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5df41ffb-123b-45fd-a3ae-3c61bcaafd5a-credential-socket\") pod \"5df41ffb-123b-45fd-a3ae-3c61bcaafd5a\" (UID: \"5df41ffb-123b-45fd-a3ae-3c61bcaafd5a\") "
Oct 30 05:26:00.460416 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:00.460409    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5df41ffb-123b-45fd-a3ae-3c61bcaafd5a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5df41ffb-123b-45fd-a3ae-3c61bcaafd5a-workload-certs\") pod \"5df41ffb-123b-45fd-a3ae-3c61bcaafd5a\" (UID: \"5df41ffb-123b-45fd-a3ae-3c61bcaafd5a\") "
Oct 30 05:26:00.460651 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:00.460446    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5df41ffb-123b-45fd-a3ae-3c61bcaafd5a-istio-token\" (UniqueName: \"kubernetes.io/projected/5df41ffb-123b-45fd-a3ae-3c61bcaafd5a-istio-token\") pod \"5df41ffb-123b-45fd-a3ae-3c61bcaafd5a\" (UID: \"5df41ffb-123b-45fd-a3ae-3c61bcaafd5a\") "
Oct 30 05:26:00.460651 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:00.460521    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5df41ffb-123b-45fd-a3ae-3c61bcaafd5a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5df41ffb-123b-45fd-a3ae-3c61bcaafd5a-workload-socket\") pod \"5df41ffb-123b-45fd-a3ae-3c61bcaafd5a\" (UID: \"5df41ffb-123b-45fd-a3ae-3c61bcaafd5a\") "
Oct 30 05:26:00.460651 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:00.460553    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5df41ffb-123b-45fd-a3ae-3c61bcaafd5a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5df41ffb-123b-45fd-a3ae-3c61bcaafd5a-istio-data\") pod \"5df41ffb-123b-45fd-a3ae-3c61bcaafd5a\" (UID: \"5df41ffb-123b-45fd-a3ae-3c61bcaafd5a\") "
Oct 30 05:26:00.460651 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:00.460598    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5df41ffb-123b-45fd-a3ae-3c61bcaafd5a-kube-api-access-4fdqm\" (UniqueName: \"kubernetes.io/projected/5df41ffb-123b-45fd-a3ae-3c61bcaafd5a-kube-api-access-4fdqm\") pod \"5df41ffb-123b-45fd-a3ae-3c61bcaafd5a\" (UID: \"5df41ffb-123b-45fd-a3ae-3c61bcaafd5a\") "
Oct 30 05:26:00.460651 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:00.460618    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5df41ffb-123b-45fd-a3ae-3c61bcaafd5a-credential-socket" pod "5df41ffb-123b-45fd-a3ae-3c61bcaafd5a" (UID: "5df41ffb-123b-45fd-a3ae-3c61bcaafd5a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:26:00.460651 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:00.460634    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5df41ffb-123b-45fd-a3ae-3c61bcaafd5a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5df41ffb-123b-45fd-a3ae-3c61bcaafd5a-istio-envoy\") pod \"5df41ffb-123b-45fd-a3ae-3c61bcaafd5a\" (UID: \"5df41ffb-123b-45fd-a3ae-3c61bcaafd5a\") "
Oct 30 05:26:00.461005 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:00.460671    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5df41ffb-123b-45fd-a3ae-3c61bcaafd5a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5df41ffb-123b-45fd-a3ae-3c61bcaafd5a-istiod-ca-cert\") pod \"5df41ffb-123b-45fd-a3ae-3c61bcaafd5a\" (UID: \"5df41ffb-123b-45fd-a3ae-3c61bcaafd5a\") "
Oct 30 05:26:00.461005 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:00.460746    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5df41ffb-123b-45fd-a3ae-3c61bcaafd5a-workload-certs" pod "5df41ffb-123b-45fd-a3ae-3c61bcaafd5a" (UID: "5df41ffb-123b-45fd-a3ae-3c61bcaafd5a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:26:00.461005 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:00.460809    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5df41ffb-123b-45fd-a3ae-3c61bcaafd5a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5df41ffb-123b-45fd-a3ae-3c61bcaafd5a-istio-podinfo\") pod \"5df41ffb-123b-45fd-a3ae-3c61bcaafd5a\" (UID: \"5df41ffb-123b-45fd-a3ae-3c61bcaafd5a\") "
Oct 30 05:26:00.461005 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:00.460818    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5df41ffb-123b-45fd-a3ae-3c61bcaafd5a-workload-socket" pod "5df41ffb-123b-45fd-a3ae-3c61bcaafd5a" (UID: "5df41ffb-123b-45fd-a3ae-3c61bcaafd5a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:26:00.461474 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:00.461447    5898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5df41ffb-123b-45fd-a3ae-3c61bcaafd5a-credential-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:26:00.461663 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:00.461476    5898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5df41ffb-123b-45fd-a3ae-3c61bcaafd5a-workload-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:26:00.461663 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:00.461490    5898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5df41ffb-123b-45fd-a3ae-3c61bcaafd5a-workload-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:26:00.461813 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:00.461687    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5df41ffb-123b-45fd-a3ae-3c61bcaafd5a-istiod-ca-cert" pod "5df41ffb-123b-45fd-a3ae-3c61bcaafd5a" (UID: "5df41ffb-123b-45fd-a3ae-3c61bcaafd5a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:26:00.462144 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:00.462105    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5df41ffb-123b-45fd-a3ae-3c61bcaafd5a-istio-data" pod "5df41ffb-123b-45fd-a3ae-3c61bcaafd5a" (UID: "5df41ffb-123b-45fd-a3ae-3c61bcaafd5a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:26:00.463724 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:00.463696    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5df41ffb-123b-45fd-a3ae-3c61bcaafd5a-istio-envoy" pod "5df41ffb-123b-45fd-a3ae-3c61bcaafd5a" (UID: "5df41ffb-123b-45fd-a3ae-3c61bcaafd5a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:26:00.465319 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:00.465282    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5df41ffb-123b-45fd-a3ae-3c61bcaafd5a-istio-token" pod "5df41ffb-123b-45fd-a3ae-3c61bcaafd5a" (UID: "5df41ffb-123b-45fd-a3ae-3c61bcaafd5a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:26:00.468160 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:00.468116    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5df41ffb-123b-45fd-a3ae-3c61bcaafd5a-istio-podinfo" pod "5df41ffb-123b-45fd-a3ae-3c61bcaafd5a" (UID: "5df41ffb-123b-45fd-a3ae-3c61bcaafd5a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:26:00.468287 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:00.468214    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5df41ffb-123b-45fd-a3ae-3c61bcaafd5a-kube-api-access-4fdqm" pod "5df41ffb-123b-45fd-a3ae-3c61bcaafd5a" (UID: "5df41ffb-123b-45fd-a3ae-3c61bcaafd5a"). InnerVolumeSpecName "kube-api-access-4fdqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:26:00.471553 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:00.471511    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:26:00.563036 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:00.563005    5898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5df41ffb-123b-45fd-a3ae-3c61bcaafd5a-istio-podinfo\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:26:00.563036 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:00.563030    5898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5df41ffb-123b-45fd-a3ae-3c61bcaafd5a-istio-token\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:26:00.563036 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:00.563039    5898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5df41ffb-123b-45fd-a3ae-3c61bcaafd5a-istio-data\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:26:00.563255 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:00.563047    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fdqm\" (UniqueName: \"kubernetes.io/projected/5df41ffb-123b-45fd-a3ae-3c61bcaafd5a-kube-api-access-4fdqm\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:26:00.563255 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:00.563057    5898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5df41ffb-123b-45fd-a3ae-3c61bcaafd5a-istio-envoy\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:26:00.563255 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:00.563066    5898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5df41ffb-123b-45fd-a3ae-3c61bcaafd5a-istiod-ca-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:26:00.593058 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:00.593024    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:26:00.733222 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:00.724829    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:26:00.967591 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:00.967553    5898 generic.go:356] "Generic (PLEG): container finished" podID="5df41ffb-123b-45fd-a3ae-3c61bcaafd5a" containerID="8432784f5544c48a0c29699cdeecfd6a240a51c9f80e367d5bb3995bc48bde10" exitCode=0
Oct 30 05:26:00.967749 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:00.967617    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242"
Oct 30 05:26:00.967749 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:00.967637    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242" event={"ID":"5df41ffb-123b-45fd-a3ae-3c61bcaafd5a","Type":"ContainerDied","Data":"8432784f5544c48a0c29699cdeecfd6a240a51c9f80e367d5bb3995bc48bde10"}
Oct 30 05:26:00.967749 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:00.967704    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242" event={"ID":"5df41ffb-123b-45fd-a3ae-3c61bcaafd5a","Type":"ContainerDied","Data":"89b92e881b70344371b2e3c371822626cad4379c79e0b92f2ab1e9c3805f2579"}
Oct 30 05:26:00.967749 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:00.967731    5898 scope.go:122] "RemoveContainer" containerID="8432784f5544c48a0c29699cdeecfd6a240a51c9f80e367d5bb3995bc48bde10"
Oct 30 05:26:00.978084 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:00.978066    5898 scope.go:122] "RemoveContainer" containerID="8432784f5544c48a0c29699cdeecfd6a240a51c9f80e367d5bb3995bc48bde10"
Oct 30 05:26:00.978393 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:26:00.978372    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8432784f5544c48a0c29699cdeecfd6a240a51c9f80e367d5bb3995bc48bde10\": container with ID starting with 8432784f5544c48a0c29699cdeecfd6a240a51c9f80e367d5bb3995bc48bde10 not found: ID does not exist" containerID="8432784f5544c48a0c29699cdeecfd6a240a51c9f80e367d5bb3995bc48bde10"
Oct 30 05:26:00.978458 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:00.978400    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8432784f5544c48a0c29699cdeecfd6a240a51c9f80e367d5bb3995bc48bde10"} err="failed to get container status \"8432784f5544c48a0c29699cdeecfd6a240a51c9f80e367d5bb3995bc48bde10\": rpc error: code = NotFound desc = could not find container \"8432784f5544c48a0c29699cdeecfd6a240a51c9f80e367d5bb3995bc48bde10\": container with ID starting with 8432784f5544c48a0c29699cdeecfd6a240a51c9f80e367d5bb3995bc48bde10 not found: ID does not exist"
Oct 30 05:26:01.020431 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:01.020366    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242"]
Oct 30 05:26:01.031402 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:01.031377    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-68b893-openshift-default-5d65c9cc49-66242"]
Oct 30 05:26:02.172923 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:02.172859    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:26:02.286727 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:02.286700    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:26:02.544047 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:02.543961    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:26:02.891473 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:02.891439    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5df41ffb-123b-45fd-a3ae-3c61bcaafd5a" path="/var/lib/kubelet/pods/5df41ffb-123b-45fd-a3ae-3c61bcaafd5a/volumes"
Oct 30 05:26:04.490133 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:04.490100    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:26:06.126516 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:06.126482    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-45607d-669bbf79bd-n7c4k"]
Oct 30 05:26:06.127195 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:06.127177    5898 cpu_manager.go:414] "containerMap: removing container" podUID="5df41ffb-123b-45fd-a3ae-3c61bcaafd5a" containerName="istio-proxy"
Oct 30 05:26:06.127257 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:06.127197    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5df41ffb-123b-45fd-a3ae-3c61bcaafd5a" containerName="istio-proxy"
Oct 30 05:26:06.127293 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:06.127285    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="5df41ffb-123b-45fd-a3ae-3c61bcaafd5a" containerName="istio-proxy"
Oct 30 05:26:06.132117 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:06.132099    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45607d-669bbf79bd-n7c4k"
Oct 30 05:26:06.150323 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:06.150301    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-45607d-669bbf79bd-n7c4k"]
Oct 30 05:26:06.170536 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:06.170506    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-45607d-serving-cert\""
Oct 30 05:26:06.173589 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:06.173544    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52ds5\" (UniqueName: \"kubernetes.io/projected/dcd3bd3a-e827-4762-97b8-5d1295c48811-kube-api-access-52ds5\") pod \"maas-api-e2e-ipp-b-45607d-669bbf79bd-n7c4k\" (UID: \"dcd3bd3a-e827-4762-97b8-5d1295c48811\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45607d-669bbf79bd-n7c4k"
Oct 30 05:26:06.173714 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:06.173686    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcd3bd3a-e827-4762-97b8-5d1295c48811-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-45607d-669bbf79bd-n7c4k\" (UID: \"dcd3bd3a-e827-4762-97b8-5d1295c48811\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45607d-669bbf79bd-n7c4k"
Oct 30 05:26:06.173878 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:06.173861    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcd3bd3a-e827-4762-97b8-5d1295c48811-metrics-certs\") pod \"maas-api-e2e-ipp-b-45607d-669bbf79bd-n7c4k\" (UID: \"dcd3bd3a-e827-4762-97b8-5d1295c48811\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45607d-669bbf79bd-n7c4k"
Oct 30 05:26:06.173995 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:06.173979    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcd3bd3a-e827-4762-97b8-5d1295c48811-maas-api-tls\") pod \"maas-api-e2e-ipp-b-45607d-669bbf79bd-n7c4k\" (UID: \"dcd3bd3a-e827-4762-97b8-5d1295c48811\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45607d-669bbf79bd-n7c4k"
Oct 30 05:26:06.213859 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:06.213830    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:26:06.274990 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:06.274954    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcd3bd3a-e827-4762-97b8-5d1295c48811-metrics-certs\") pod \"maas-api-e2e-ipp-b-45607d-669bbf79bd-n7c4k\" (UID: \"dcd3bd3a-e827-4762-97b8-5d1295c48811\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45607d-669bbf79bd-n7c4k"
Oct 30 05:26:06.275168 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:06.275052    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcd3bd3a-e827-4762-97b8-5d1295c48811-maas-api-tls\") pod \"maas-api-e2e-ipp-b-45607d-669bbf79bd-n7c4k\" (UID: \"dcd3bd3a-e827-4762-97b8-5d1295c48811\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45607d-669bbf79bd-n7c4k"
Oct 30 05:26:06.275168 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:06.275146    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52ds5\" (UniqueName: \"kubernetes.io/projected/dcd3bd3a-e827-4762-97b8-5d1295c48811-kube-api-access-52ds5\") pod \"maas-api-e2e-ipp-b-45607d-669bbf79bd-n7c4k\" (UID: \"dcd3bd3a-e827-4762-97b8-5d1295c48811\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45607d-669bbf79bd-n7c4k"
Oct 30 05:26:06.275256 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:06.275210    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcd3bd3a-e827-4762-97b8-5d1295c48811-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-45607d-669bbf79bd-n7c4k\" (UID: \"dcd3bd3a-e827-4762-97b8-5d1295c48811\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45607d-669bbf79bd-n7c4k"
Oct 30 05:26:06.276016 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:06.275988    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcd3bd3a-e827-4762-97b8-5d1295c48811-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-45607d-669bbf79bd-n7c4k\" (UID: \"dcd3bd3a-e827-4762-97b8-5d1295c48811\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45607d-669bbf79bd-n7c4k"
Oct 30 05:26:06.277675 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:06.277648    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcd3bd3a-e827-4762-97b8-5d1295c48811-metrics-certs\") pod \"maas-api-e2e-ipp-b-45607d-669bbf79bd-n7c4k\" (UID: \"dcd3bd3a-e827-4762-97b8-5d1295c48811\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45607d-669bbf79bd-n7c4k"
Oct 30 05:26:06.277804 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:06.277723    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcd3bd3a-e827-4762-97b8-5d1295c48811-maas-api-tls\") pod \"maas-api-e2e-ipp-b-45607d-669bbf79bd-n7c4k\" (UID: \"dcd3bd3a-e827-4762-97b8-5d1295c48811\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45607d-669bbf79bd-n7c4k"
Oct 30 05:26:06.294068 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:06.294043    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52ds5\" (UniqueName: \"kubernetes.io/projected/dcd3bd3a-e827-4762-97b8-5d1295c48811-kube-api-access-52ds5\") pod \"maas-api-e2e-ipp-b-45607d-669bbf79bd-n7c4k\" (UID: \"dcd3bd3a-e827-4762-97b8-5d1295c48811\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45607d-669bbf79bd-n7c4k"
Oct 30 05:26:06.318432 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:06.318396    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:26:06.445808 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:06.445726    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45607d-669bbf79bd-n7c4k"
Oct 30 05:26:06.750131 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:06.750105    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-45607d-669bbf79bd-n7c4k"]
Oct 30 05:26:06.753022 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:26:06.752990    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcd3bd3a_e827_4762_97b8_5d1295c48811.slice/crio-8d326ef6f772e216d89d8857c7dd401fdbbc9c4fb8f7dbe2a51b1b4d8cf59fb6 WatchSource:0}: Error finding container 8d326ef6f772e216d89d8857c7dd401fdbbc9c4fb8f7dbe2a51b1b4d8cf59fb6: Status 404 returned error can't find the container with id 8d326ef6f772e216d89d8857c7dd401fdbbc9c4fb8f7dbe2a51b1b4d8cf59fb6
Oct 30 05:26:06.755235 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:06.755213    5898 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 05:26:07.012593 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:07.012513    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45607d-669bbf79bd-n7c4k" event={"ID":"dcd3bd3a-e827-4762-97b8-5d1295c48811","Type":"ContainerStarted","Data":"8d326ef6f772e216d89d8857c7dd401fdbbc9c4fb8f7dbe2a51b1b4d8cf59fb6"}
Oct 30 05:26:08.020420 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:08.020380    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45607d-669bbf79bd-n7c4k" event={"ID":"dcd3bd3a-e827-4762-97b8-5d1295c48811","Type":"ContainerStarted","Data":"3ea304aacde8642a948c17795f82e7bb5bd90e4355036ddbc5be0c74bb8adbfd"}
Oct 30 05:26:08.020824 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:08.020463    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45607d-669bbf79bd-n7c4k"
Oct 30 05:26:08.061173 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:08.061121    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45607d-669bbf79bd-n7c4k" podStartSLOduration=1.739349314 podStartE2EDuration="2.061105063s" totalImagesPullingTime="321.755749ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:26:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:26:08.060974632 +0000 UTC m=+2497.284794533" watchObservedRunningTime="2026-10-30 05:26:08.061105063 +0000 UTC m=+2497.284924964"
Oct 30 05:26:08.083729 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:08.083700    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:26:09.982383 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:09.982350    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:26:11.678805 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:11.678772    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:26:13.551218 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:13.551183    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:26:14.032926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:14.032885    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45607d-669bbf79bd-n7c4k"
Oct 30 05:26:15.620436 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:15.620406    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:26:17.362209 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:17.362163    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:26:17.803827 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:17.803705    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:26:19.272335 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:19.272267    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:26:20.953149 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:20.953109    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:26:22.709062 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:22.709022    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:26:24.596165 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:24.596127    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:26:26.803494 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:26.803457    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:26:26.821364 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:26:26.821325    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:26:28.591108 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:28.591080    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:26:30.358318 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:30.358283    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:26:31.455427 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:31.455397    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:26:31.562984 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:31.562950    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:26:31.663283 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:31.663252    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:26:32.191947 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:32.191888    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:26:33.911096 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:33.911064    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:26:35.572633 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:35.572597    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:26:37.470973 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:37.470939    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:26:39.102149 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:39.102113    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:26:40.417325 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:40.417291    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:26:40.864879 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:40.864848    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:26:42.477864 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:42.477754    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:26:42.745556 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:42.745458    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:26:42.862958 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:42.861493    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:26:42.960265 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:42.960232    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:26:44.596180 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:44.596144    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:26:46.344217 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:46.344183    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:26:48.253680 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:48.253645    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:26:48.828071 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:48.828034    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:26:49.926459 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:49.926414    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:26:50.135944 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:50.132817    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:26:51.893135 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:51.893099    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:26:53.713080 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:53.713047    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:26:55.059397 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:55.059363    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:26:55.167922 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:55.164646    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:26:55.265800 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:55.265762    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:26:55.729387 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:55.729352    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:26:56.819419 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:26:56.819381    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:26:57.498634 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:57.498566    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:26:59.233099 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:26:59.233066    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:27:01.093647 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:01.093613    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:27:02.743246 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:02.743211    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:27:04.073928 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:04.073684    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:27:04.518856 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:04.518781    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:27:06.617166 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:06.617135    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:27:08.590626 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:08.590586    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:27:10.952752 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:10.952721    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:27:12.344049 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:12.344016    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:27:12.571850 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:12.571698    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:27:13.586417 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:13.586381    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:27:15.137404 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:15.137061    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:27:15.392509 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:15.392429    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:27:17.287965 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:17.287934    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:27:18.943877 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:18.943844    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:27:19.167643 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:19.167608    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d"]
Oct 30 05:27:19.172083 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:19.172062    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d"
Oct 30 05:27:19.196701 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:19.196638    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d"]
Oct 30 05:27:19.269864 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:19.269829    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-45607d-openshift-default-dockercfg-dlgvx\""
Oct 30 05:27:19.289670 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:19.289639    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/572f0bd4-2d12-4bcb-adce-edf0c3926179-istio-token\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d\" (UID: \"572f0bd4-2d12-4bcb-adce-edf0c3926179\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d"
Oct 30 05:27:19.289832 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:19.289710    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/572f0bd4-2d12-4bcb-adce-edf0c3926179-istio-data\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d\" (UID: \"572f0bd4-2d12-4bcb-adce-edf0c3926179\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d"
Oct 30 05:27:19.289832 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:19.289814    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/572f0bd4-2d12-4bcb-adce-edf0c3926179-workload-certs\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d\" (UID: \"572f0bd4-2d12-4bcb-adce-edf0c3926179\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d"
Oct 30 05:27:19.289947 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:19.289854    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/572f0bd4-2d12-4bcb-adce-edf0c3926179-istio-envoy\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d\" (UID: \"572f0bd4-2d12-4bcb-adce-edf0c3926179\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d"
Oct 30 05:27:19.289947 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:19.289914    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxhn6\" (UniqueName: \"kubernetes.io/projected/572f0bd4-2d12-4bcb-adce-edf0c3926179-kube-api-access-jxhn6\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d\" (UID: \"572f0bd4-2d12-4bcb-adce-edf0c3926179\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d"
Oct 30 05:27:19.290030 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:19.289985    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/572f0bd4-2d12-4bcb-adce-edf0c3926179-istio-podinfo\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d\" (UID: \"572f0bd4-2d12-4bcb-adce-edf0c3926179\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d"
Oct 30 05:27:19.290066 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:19.290036    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/572f0bd4-2d12-4bcb-adce-edf0c3926179-istiod-ca-cert\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d\" (UID: \"572f0bd4-2d12-4bcb-adce-edf0c3926179\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d"
Oct 30 05:27:19.290175 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:19.290152    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/572f0bd4-2d12-4bcb-adce-edf0c3926179-workload-socket\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d\" (UID: \"572f0bd4-2d12-4bcb-adce-edf0c3926179\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d"
Oct 30 05:27:19.290269 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:19.290256    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/572f0bd4-2d12-4bcb-adce-edf0c3926179-credential-socket\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d\" (UID: \"572f0bd4-2d12-4bcb-adce-edf0c3926179\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d"
Oct 30 05:27:19.391324 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:19.391285    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/572f0bd4-2d12-4bcb-adce-edf0c3926179-workload-socket\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d\" (UID: \"572f0bd4-2d12-4bcb-adce-edf0c3926179\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d"
Oct 30 05:27:19.391526 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:19.391378    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/572f0bd4-2d12-4bcb-adce-edf0c3926179-credential-socket\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d\" (UID: \"572f0bd4-2d12-4bcb-adce-edf0c3926179\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d"
Oct 30 05:27:19.391526 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:19.391423    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/572f0bd4-2d12-4bcb-adce-edf0c3926179-istio-token\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d\" (UID: \"572f0bd4-2d12-4bcb-adce-edf0c3926179\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d"
Oct 30 05:27:19.391526 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:19.391505    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/572f0bd4-2d12-4bcb-adce-edf0c3926179-istio-data\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d\" (UID: \"572f0bd4-2d12-4bcb-adce-edf0c3926179\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d"
Oct 30 05:27:19.392090 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:19.391561    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/572f0bd4-2d12-4bcb-adce-edf0c3926179-workload-certs\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d\" (UID: \"572f0bd4-2d12-4bcb-adce-edf0c3926179\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d"
Oct 30 05:27:19.392090 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:19.391625    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/572f0bd4-2d12-4bcb-adce-edf0c3926179-istio-envoy\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d\" (UID: \"572f0bd4-2d12-4bcb-adce-edf0c3926179\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d"
Oct 30 05:27:19.392090 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:19.391661    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxhn6\" (UniqueName: \"kubernetes.io/projected/572f0bd4-2d12-4bcb-adce-edf0c3926179-kube-api-access-jxhn6\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d\" (UID: \"572f0bd4-2d12-4bcb-adce-edf0c3926179\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d"
Oct 30 05:27:19.392090 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:19.391694    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/572f0bd4-2d12-4bcb-adce-edf0c3926179-workload-socket\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d\" (UID: \"572f0bd4-2d12-4bcb-adce-edf0c3926179\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d"
Oct 30 05:27:19.392090 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:19.391698    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/572f0bd4-2d12-4bcb-adce-edf0c3926179-istio-podinfo\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d\" (UID: \"572f0bd4-2d12-4bcb-adce-edf0c3926179\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d"
Oct 30 05:27:19.392090 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:19.391772    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/572f0bd4-2d12-4bcb-adce-edf0c3926179-credential-socket\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d\" (UID: \"572f0bd4-2d12-4bcb-adce-edf0c3926179\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d"
Oct 30 05:27:19.392090 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:19.391814    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/572f0bd4-2d12-4bcb-adce-edf0c3926179-istiod-ca-cert\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d\" (UID: \"572f0bd4-2d12-4bcb-adce-edf0c3926179\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d"
Oct 30 05:27:19.392090 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:19.391928    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/572f0bd4-2d12-4bcb-adce-edf0c3926179-workload-certs\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d\" (UID: \"572f0bd4-2d12-4bcb-adce-edf0c3926179\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d"
Oct 30 05:27:19.392090 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:19.391944    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/572f0bd4-2d12-4bcb-adce-edf0c3926179-istio-data\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d\" (UID: \"572f0bd4-2d12-4bcb-adce-edf0c3926179\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d"
Oct 30 05:27:19.392493 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:19.392400    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/572f0bd4-2d12-4bcb-adce-edf0c3926179-istiod-ca-cert\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d\" (UID: \"572f0bd4-2d12-4bcb-adce-edf0c3926179\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d"
Oct 30 05:27:19.393938 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:19.393916    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/572f0bd4-2d12-4bcb-adce-edf0c3926179-istio-envoy\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d\" (UID: \"572f0bd4-2d12-4bcb-adce-edf0c3926179\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d"
Oct 30 05:27:19.394242 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:19.394222    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/572f0bd4-2d12-4bcb-adce-edf0c3926179-istio-podinfo\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d\" (UID: \"572f0bd4-2d12-4bcb-adce-edf0c3926179\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d"
Oct 30 05:27:19.410051 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:19.410026    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/572f0bd4-2d12-4bcb-adce-edf0c3926179-istio-token\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d\" (UID: \"572f0bd4-2d12-4bcb-adce-edf0c3926179\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d"
Oct 30 05:27:19.410186 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:19.410164    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxhn6\" (UniqueName: \"kubernetes.io/projected/572f0bd4-2d12-4bcb-adce-edf0c3926179-kube-api-access-jxhn6\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d\" (UID: \"572f0bd4-2d12-4bcb-adce-edf0c3926179\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d"
Oct 30 05:27:19.487504 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:19.487429    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d"
Oct 30 05:27:19.818082 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:19.818029    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d"]
Oct 30 05:27:19.819829 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:27:19.819800    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod572f0bd4_2d12_4bcb_adce_edf0c3926179.slice/crio-aaf7bd51bedb39f040a68b04140f9fba53a3a4bb47ec603ba4f4eae36b205e73 WatchSource:0}: Error finding container aaf7bd51bedb39f040a68b04140f9fba53a3a4bb47ec603ba4f4eae36b205e73: Status 404 returned error can't find the container with id aaf7bd51bedb39f040a68b04140f9fba53a3a4bb47ec603ba4f4eae36b205e73
Oct 30 05:27:19.823329 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:19.823288    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:27:19.823418 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:19.823379    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:27:19.823464 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:19.823419    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:27:20.641677 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:20.641626    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d" event={"ID":"572f0bd4-2d12-4bcb-adce-edf0c3926179","Type":"ContainerStarted","Data":"ab62b30ea1c501479ed1b4c44fbd0a12657a9d555f28f355fb57561da7b0ac77"}
Oct 30 05:27:20.642218 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:20.641725    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d" event={"ID":"572f0bd4-2d12-4bcb-adce-edf0c3926179","Type":"ContainerStarted","Data":"aaf7bd51bedb39f040a68b04140f9fba53a3a4bb47ec603ba4f4eae36b205e73"}
Oct 30 05:27:20.692012 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:20.691965    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d" podStartSLOduration=1.6919499230000001 podStartE2EDuration="1.691949923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:27:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:27:20.69117646 +0000 UTC m=+2569.914996362" watchObservedRunningTime="2026-10-30 05:27:20.691949923 +0000 UTC m=+2569.915769825"
Oct 30 05:27:20.719164 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:20.719135    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:27:21.491923 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:21.491307    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d"
Oct 30 05:27:21.500072 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:21.500032    5898 patch_prober.go:29] interesting pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Oct 30 05:27:21.500277 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:21.500097    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d" podUID="572f0bd4-2d12-4bcb-adce-edf0c3926179" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Oct 30 05:27:22.491924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:22.491262    5898 patch_prober.go:29] interesting pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Oct 30 05:27:22.491924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:22.491326    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d" podUID="572f0bd4-2d12-4bcb-adce-edf0c3926179" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Oct 30 05:27:22.787986 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:22.787504    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:27:23.490923 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:23.490266    5898 patch_prober.go:29] interesting pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Oct 30 05:27:23.490923 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:23.490333    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d" podUID="572f0bd4-2d12-4bcb-adce-edf0c3926179" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Oct 30 05:27:24.310131 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:24.310096    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-93bad9/rate-test-model-93bad9-kserve-6878b4bf49-hcjsm"]
Oct 30 05:27:24.314605 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:24.314580    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-93bad9/rate-test-model-93bad9-kserve-6878b4bf49-hcjsm"
Oct 30 05:27:24.336747 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:24.336710    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-93bad9/rate-test-model-93bad9-kserve-6878b4bf49-hcjsm"]
Oct 30 05:27:24.340499 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:24.339671    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-93bad9\"/\"default-dockercfg-h552l\""
Oct 30 05:27:24.340499 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:24.339744    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-93bad9\"/\"rate-test-model-93bad9-kserve-self-signed-certs\""
Oct 30 05:27:24.341791 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:24.341776    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-93bad9\"/\"openshift-service-ca.crt\""
Oct 30 05:27:24.342180 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:24.342130    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-93bad9\"/\"kube-root-ca.crt\""
Oct 30 05:27:24.371727 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:24.371704    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:27:24.463809 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:24.463773    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66cbc6ec-813b-4904-b65e-8893ae4f02ce-dshm\") pod \"rate-test-model-93bad9-kserve-6878b4bf49-hcjsm\" (UID: \"66cbc6ec-813b-4904-b65e-8893ae4f02ce\") " pod="ai-tenant-e2e-shared-b-w2-93bad9/rate-test-model-93bad9-kserve-6878b4bf49-hcjsm"
Oct 30 05:27:24.464026 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:24.463850    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g8xj\" (UniqueName: \"kubernetes.io/projected/66cbc6ec-813b-4904-b65e-8893ae4f02ce-kube-api-access-4g8xj\") pod \"rate-test-model-93bad9-kserve-6878b4bf49-hcjsm\" (UID: \"66cbc6ec-813b-4904-b65e-8893ae4f02ce\") " pod="ai-tenant-e2e-shared-b-w2-93bad9/rate-test-model-93bad9-kserve-6878b4bf49-hcjsm"
Oct 30 05:27:24.464094 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:24.464031    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66cbc6ec-813b-4904-b65e-8893ae4f02ce-tls-certs\") pod \"rate-test-model-93bad9-kserve-6878b4bf49-hcjsm\" (UID: \"66cbc6ec-813b-4904-b65e-8893ae4f02ce\") " pod="ai-tenant-e2e-shared-b-w2-93bad9/rate-test-model-93bad9-kserve-6878b4bf49-hcjsm"
Oct 30 05:27:24.464094 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:24.464074    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66cbc6ec-813b-4904-b65e-8893ae4f02ce-home\") pod \"rate-test-model-93bad9-kserve-6878b4bf49-hcjsm\" (UID: \"66cbc6ec-813b-4904-b65e-8893ae4f02ce\") " pod="ai-tenant-e2e-shared-b-w2-93bad9/rate-test-model-93bad9-kserve-6878b4bf49-hcjsm"
Oct 30 05:27:24.464207 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:24.464191    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66cbc6ec-813b-4904-b65e-8893ae4f02ce-tmp-dir\") pod \"rate-test-model-93bad9-kserve-6878b4bf49-hcjsm\" (UID: \"66cbc6ec-813b-4904-b65e-8893ae4f02ce\") " pod="ai-tenant-e2e-shared-b-w2-93bad9/rate-test-model-93bad9-kserve-6878b4bf49-hcjsm"
Oct 30 05:27:24.464267 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:24.464251    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66cbc6ec-813b-4904-b65e-8893ae4f02ce-model-cache\") pod \"rate-test-model-93bad9-kserve-6878b4bf49-hcjsm\" (UID: \"66cbc6ec-813b-4904-b65e-8893ae4f02ce\") " pod="ai-tenant-e2e-shared-b-w2-93bad9/rate-test-model-93bad9-kserve-6878b4bf49-hcjsm"
Oct 30 05:27:24.489925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:24.488243    5898 patch_prober.go:29] interesting pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Oct 30 05:27:24.489925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:24.488300    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d" podUID="572f0bd4-2d12-4bcb-adce-edf0c3926179" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Oct 30 05:27:24.565533 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:24.565435    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66cbc6ec-813b-4904-b65e-8893ae4f02ce-dshm\") pod \"rate-test-model-93bad9-kserve-6878b4bf49-hcjsm\" (UID: \"66cbc6ec-813b-4904-b65e-8893ae4f02ce\") " pod="ai-tenant-e2e-shared-b-w2-93bad9/rate-test-model-93bad9-kserve-6878b4bf49-hcjsm"
Oct 30 05:27:24.565533 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:24.565491    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g8xj\" (UniqueName: \"kubernetes.io/projected/66cbc6ec-813b-4904-b65e-8893ae4f02ce-kube-api-access-4g8xj\") pod \"rate-test-model-93bad9-kserve-6878b4bf49-hcjsm\" (UID: \"66cbc6ec-813b-4904-b65e-8893ae4f02ce\") " pod="ai-tenant-e2e-shared-b-w2-93bad9/rate-test-model-93bad9-kserve-6878b4bf49-hcjsm"
Oct 30 05:27:24.565774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:24.565548    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66cbc6ec-813b-4904-b65e-8893ae4f02ce-tls-certs\") pod \"rate-test-model-93bad9-kserve-6878b4bf49-hcjsm\" (UID: \"66cbc6ec-813b-4904-b65e-8893ae4f02ce\") " pod="ai-tenant-e2e-shared-b-w2-93bad9/rate-test-model-93bad9-kserve-6878b4bf49-hcjsm"
Oct 30 05:27:24.565774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:24.565565    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66cbc6ec-813b-4904-b65e-8893ae4f02ce-home\") pod \"rate-test-model-93bad9-kserve-6878b4bf49-hcjsm\" (UID: \"66cbc6ec-813b-4904-b65e-8893ae4f02ce\") " pod="ai-tenant-e2e-shared-b-w2-93bad9/rate-test-model-93bad9-kserve-6878b4bf49-hcjsm"
Oct 30 05:27:24.565774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:24.565616    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66cbc6ec-813b-4904-b65e-8893ae4f02ce-tmp-dir\") pod \"rate-test-model-93bad9-kserve-6878b4bf49-hcjsm\" (UID: \"66cbc6ec-813b-4904-b65e-8893ae4f02ce\") " pod="ai-tenant-e2e-shared-b-w2-93bad9/rate-test-model-93bad9-kserve-6878b4bf49-hcjsm"
Oct 30 05:27:24.565774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:24.565639    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66cbc6ec-813b-4904-b65e-8893ae4f02ce-model-cache\") pod \"rate-test-model-93bad9-kserve-6878b4bf49-hcjsm\" (UID: \"66cbc6ec-813b-4904-b65e-8893ae4f02ce\") " pod="ai-tenant-e2e-shared-b-w2-93bad9/rate-test-model-93bad9-kserve-6878b4bf49-hcjsm"
Oct 30 05:27:24.566050 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:24.566026    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66cbc6ec-813b-4904-b65e-8893ae4f02ce-model-cache\") pod \"rate-test-model-93bad9-kserve-6878b4bf49-hcjsm\" (UID: \"66cbc6ec-813b-4904-b65e-8893ae4f02ce\") " pod="ai-tenant-e2e-shared-b-w2-93bad9/rate-test-model-93bad9-kserve-6878b4bf49-hcjsm"
Oct 30 05:27:24.566831 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:24.566695    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66cbc6ec-813b-4904-b65e-8893ae4f02ce-home\") pod \"rate-test-model-93bad9-kserve-6878b4bf49-hcjsm\" (UID: \"66cbc6ec-813b-4904-b65e-8893ae4f02ce\") " pod="ai-tenant-e2e-shared-b-w2-93bad9/rate-test-model-93bad9-kserve-6878b4bf49-hcjsm"
Oct 30 05:27:24.567035 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:24.567011    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66cbc6ec-813b-4904-b65e-8893ae4f02ce-tmp-dir\") pod \"rate-test-model-93bad9-kserve-6878b4bf49-hcjsm\" (UID: \"66cbc6ec-813b-4904-b65e-8893ae4f02ce\") " pod="ai-tenant-e2e-shared-b-w2-93bad9/rate-test-model-93bad9-kserve-6878b4bf49-hcjsm"
Oct 30 05:27:24.568977 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:24.568948    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66cbc6ec-813b-4904-b65e-8893ae4f02ce-dshm\") pod \"rate-test-model-93bad9-kserve-6878b4bf49-hcjsm\" (UID: \"66cbc6ec-813b-4904-b65e-8893ae4f02ce\") " pod="ai-tenant-e2e-shared-b-w2-93bad9/rate-test-model-93bad9-kserve-6878b4bf49-hcjsm"
Oct 30 05:27:24.569845 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:24.569821    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66cbc6ec-813b-4904-b65e-8893ae4f02ce-tls-certs\") pod \"rate-test-model-93bad9-kserve-6878b4bf49-hcjsm\" (UID: \"66cbc6ec-813b-4904-b65e-8893ae4f02ce\") " pod="ai-tenant-e2e-shared-b-w2-93bad9/rate-test-model-93bad9-kserve-6878b4bf49-hcjsm"
Oct 30 05:27:24.616925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:24.595849    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g8xj\" (UniqueName: \"kubernetes.io/projected/66cbc6ec-813b-4904-b65e-8893ae4f02ce-kube-api-access-4g8xj\") pod \"rate-test-model-93bad9-kserve-6878b4bf49-hcjsm\" (UID: \"66cbc6ec-813b-4904-b65e-8893ae4f02ce\") " pod="ai-tenant-e2e-shared-b-w2-93bad9/rate-test-model-93bad9-kserve-6878b4bf49-hcjsm"
Oct 30 05:27:24.644084 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:24.644045    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-93bad9/rate-test-model-93bad9-kserve-6878b4bf49-hcjsm"
Oct 30 05:27:24.670032 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:24.669950    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:27:25.204120 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:25.204091    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-93bad9/rate-test-model-93bad9-kserve-6878b4bf49-hcjsm"]
Oct 30 05:27:25.205023 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:27:25.204998    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod66cbc6ec_813b_4904_b65e_8893ae4f02ce.slice/crio-322f8739327de730b4121a03d172438a0e2116286c7cb0a598402e41e5fcba59 WatchSource:0}: Error finding container 322f8739327de730b4121a03d172438a0e2116286c7cb0a598402e41e5fcba59: Status 404 returned error can't find the container with id 322f8739327de730b4121a03d172438a0e2116286c7cb0a598402e41e5fcba59
Oct 30 05:27:25.491785 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:25.491271    5898 patch_prober.go:29] interesting pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Oct 30 05:27:25.491785 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:25.491333    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d" podUID="572f0bd4-2d12-4bcb-adce-edf0c3926179" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Oct 30 05:27:25.707807 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:25.707760    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-93bad9/rate-test-model-93bad9-kserve-6878b4bf49-hcjsm" event={"ID":"66cbc6ec-813b-4904-b65e-8893ae4f02ce","Type":"ContainerStarted","Data":"3b76e11779a00c3019d5462c652552f2c1ef734c7f04cda9587a07f1bf782835"}
Oct 30 05:27:25.707996 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:25.707819    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-93bad9/rate-test-model-93bad9-kserve-6878b4bf49-hcjsm" event={"ID":"66cbc6ec-813b-4904-b65e-8893ae4f02ce","Type":"ContainerStarted","Data":"322f8739327de730b4121a03d172438a0e2116286c7cb0a598402e41e5fcba59"}
Oct 30 05:27:25.748815 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:25.748719    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-93bad9/rate-test-model-93bad9-kserve-6878b4bf49-hcjsm" podStartSLOduration=1.7486992909999999 podStartE2EDuration="1.748699291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:27:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:27:25.746596024 +0000 UTC m=+2574.970415935" watchObservedRunningTime="2026-10-30 05:27:25.748699291 +0000 UTC m=+2574.972519192"
Oct 30 05:27:26.525387 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:26.525351    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d"
Oct 30 05:27:26.526281 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:26.526258    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d"
Oct 30 05:27:26.527712 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:26.527693    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d"
Oct 30 05:27:26.563675 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:26.563649    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:27:26.644747 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:26.644717    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-93bad9/rate-test-model-93bad9-kserve-6878b4bf49-hcjsm"
Oct 30 05:27:26.731665 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:26.731631    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85"]
Oct 30 05:27:26.767394 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:26.767236    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85"]
Oct 30 05:27:26.767394 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:26.767364    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85"
Oct 30 05:27:26.837063 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:27:26.837024    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:27:26.842773 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:26.842742    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-35a94e-openshift-default-dockercfg-s9msl\""
Oct 30 05:27:26.927840 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:26.927811    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9f26\" (UniqueName: \"kubernetes.io/projected/96f0bc26-fbf5-4d4c-a146-207dc12a4e20-kube-api-access-f9f26\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85\" (UID: \"96f0bc26-fbf5-4d4c-a146-207dc12a4e20\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85"
Oct 30 05:27:26.928029 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:26.927959    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96f0bc26-fbf5-4d4c-a146-207dc12a4e20-credential-socket\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85\" (UID: \"96f0bc26-fbf5-4d4c-a146-207dc12a4e20\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85"
Oct 30 05:27:26.928089 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:26.928030    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96f0bc26-fbf5-4d4c-a146-207dc12a4e20-istio-envoy\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85\" (UID: \"96f0bc26-fbf5-4d4c-a146-207dc12a4e20\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85"
Oct 30 05:27:26.928089 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:26.928048    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96f0bc26-fbf5-4d4c-a146-207dc12a4e20-istio-podinfo\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85\" (UID: \"96f0bc26-fbf5-4d4c-a146-207dc12a4e20\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85"
Oct 30 05:27:26.928190 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:26.928172    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96f0bc26-fbf5-4d4c-a146-207dc12a4e20-istio-token\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85\" (UID: \"96f0bc26-fbf5-4d4c-a146-207dc12a4e20\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85"
Oct 30 05:27:26.928290 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:26.928274    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96f0bc26-fbf5-4d4c-a146-207dc12a4e20-workload-certs\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85\" (UID: \"96f0bc26-fbf5-4d4c-a146-207dc12a4e20\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85"
Oct 30 05:27:26.928363 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:26.928341    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96f0bc26-fbf5-4d4c-a146-207dc12a4e20-istiod-ca-cert\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85\" (UID: \"96f0bc26-fbf5-4d4c-a146-207dc12a4e20\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85"
Oct 30 05:27:26.928428 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:26.928400    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96f0bc26-fbf5-4d4c-a146-207dc12a4e20-istio-data\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85\" (UID: \"96f0bc26-fbf5-4d4c-a146-207dc12a4e20\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85"
Oct 30 05:27:26.928486 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:26.928432    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96f0bc26-fbf5-4d4c-a146-207dc12a4e20-workload-socket\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85\" (UID: \"96f0bc26-fbf5-4d4c-a146-207dc12a4e20\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85"
Oct 30 05:27:27.030191 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:27.030158    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9f26\" (UniqueName: \"kubernetes.io/projected/96f0bc26-fbf5-4d4c-a146-207dc12a4e20-kube-api-access-f9f26\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85\" (UID: \"96f0bc26-fbf5-4d4c-a146-207dc12a4e20\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85"
Oct 30 05:27:27.030356 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:27.030268    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96f0bc26-fbf5-4d4c-a146-207dc12a4e20-credential-socket\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85\" (UID: \"96f0bc26-fbf5-4d4c-a146-207dc12a4e20\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85"
Oct 30 05:27:27.030356 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:27.030345    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96f0bc26-fbf5-4d4c-a146-207dc12a4e20-istio-envoy\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85\" (UID: \"96f0bc26-fbf5-4d4c-a146-207dc12a4e20\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85"
Oct 30 05:27:27.030445 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:27.030371    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96f0bc26-fbf5-4d4c-a146-207dc12a4e20-istio-podinfo\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85\" (UID: \"96f0bc26-fbf5-4d4c-a146-207dc12a4e20\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85"
Oct 30 05:27:27.030445 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:27.030430    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96f0bc26-fbf5-4d4c-a146-207dc12a4e20-istio-token\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85\" (UID: \"96f0bc26-fbf5-4d4c-a146-207dc12a4e20\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85"
Oct 30 05:27:27.030542 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:27.030525    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96f0bc26-fbf5-4d4c-a146-207dc12a4e20-workload-certs\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85\" (UID: \"96f0bc26-fbf5-4d4c-a146-207dc12a4e20\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85"
Oct 30 05:27:27.030599 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:27.030550    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96f0bc26-fbf5-4d4c-a146-207dc12a4e20-istiod-ca-cert\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85\" (UID: \"96f0bc26-fbf5-4d4c-a146-207dc12a4e20\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85"
Oct 30 05:27:27.030599 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:27.030590    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96f0bc26-fbf5-4d4c-a146-207dc12a4e20-istio-data\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85\" (UID: \"96f0bc26-fbf5-4d4c-a146-207dc12a4e20\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85"
Oct 30 05:27:27.030717 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:27.030620    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96f0bc26-fbf5-4d4c-a146-207dc12a4e20-workload-socket\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85\" (UID: \"96f0bc26-fbf5-4d4c-a146-207dc12a4e20\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85"
Oct 30 05:27:27.030717 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:27.030682    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96f0bc26-fbf5-4d4c-a146-207dc12a4e20-credential-socket\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85\" (UID: \"96f0bc26-fbf5-4d4c-a146-207dc12a4e20\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85"
Oct 30 05:27:27.030947 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:27.030892    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96f0bc26-fbf5-4d4c-a146-207dc12a4e20-workload-certs\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85\" (UID: \"96f0bc26-fbf5-4d4c-a146-207dc12a4e20\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85"
Oct 30 05:27:27.031111 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:27.031009    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96f0bc26-fbf5-4d4c-a146-207dc12a4e20-istio-data\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85\" (UID: \"96f0bc26-fbf5-4d4c-a146-207dc12a4e20\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85"
Oct 30 05:27:27.031111 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:27.031058    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96f0bc26-fbf5-4d4c-a146-207dc12a4e20-workload-socket\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85\" (UID: \"96f0bc26-fbf5-4d4c-a146-207dc12a4e20\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85"
Oct 30 05:27:27.031385 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:27.031364    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96f0bc26-fbf5-4d4c-a146-207dc12a4e20-istiod-ca-cert\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85\" (UID: \"96f0bc26-fbf5-4d4c-a146-207dc12a4e20\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85"
Oct 30 05:27:27.032655 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:27.032636    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96f0bc26-fbf5-4d4c-a146-207dc12a4e20-istio-envoy\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85\" (UID: \"96f0bc26-fbf5-4d4c-a146-207dc12a4e20\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85"
Oct 30 05:27:27.032871 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:27.032853    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96f0bc26-fbf5-4d4c-a146-207dc12a4e20-istio-podinfo\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85\" (UID: \"96f0bc26-fbf5-4d4c-a146-207dc12a4e20\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85"
Oct 30 05:27:27.052013 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:27.051957    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9f26\" (UniqueName: \"kubernetes.io/projected/96f0bc26-fbf5-4d4c-a146-207dc12a4e20-kube-api-access-f9f26\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85\" (UID: \"96f0bc26-fbf5-4d4c-a146-207dc12a4e20\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85"
Oct 30 05:27:27.052369 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:27.052350    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96f0bc26-fbf5-4d4c-a146-207dc12a4e20-istio-token\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85\" (UID: \"96f0bc26-fbf5-4d4c-a146-207dc12a4e20\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85"
Oct 30 05:27:27.103675 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:27.103647    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85"
Oct 30 05:27:27.422724 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:27.422698    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85"]
Oct 30 05:27:27.425653 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:27:27.425623    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96f0bc26_fbf5_4d4c_a146_207dc12a4e20.slice/crio-8a6c8bbb55d8e7ab076ba6588f13424624eed042e1684d8f0c134a712e34a544 WatchSource:0}: Error finding container 8a6c8bbb55d8e7ab076ba6588f13424624eed042e1684d8f0c134a712e34a544: Status 404 returned error can't find the container with id 8a6c8bbb55d8e7ab076ba6588f13424624eed042e1684d8f0c134a712e34a544
Oct 30 05:27:27.428693 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:27.428660    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:27:27.428791 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:27.428743    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:27:27.428791 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:27.428771    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:27:27.725837 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:27.725749    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85" event={"ID":"96f0bc26-fbf5-4d4c-a146-207dc12a4e20","Type":"ContainerStarted","Data":"9eeba5e3797d862b575859206e238d2cb92adc2465f585531673e335198cec8b"}
Oct 30 05:27:27.725837 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:27.725806    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85" event={"ID":"96f0bc26-fbf5-4d4c-a146-207dc12a4e20","Type":"ContainerStarted","Data":"8a6c8bbb55d8e7ab076ba6588f13424624eed042e1684d8f0c134a712e34a544"}
Oct 30 05:27:27.795181 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:27.795131    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85" podStartSLOduration=1.795115963 podStartE2EDuration="1.795115963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:27:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:27:27.781221177 +0000 UTC m=+2577.005041078" watchObservedRunningTime="2026-10-30 05:27:27.795115963 +0000 UTC m=+2577.018935863"
Oct 30 05:27:28.103863 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:28.103830    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85"
Oct 30 05:27:28.105536 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:28.105509    5898 patch_prober.go:29] interesting pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Oct 30 05:27:28.105651 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:28.105564    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85" podUID="96f0bc26-fbf5-4d4c-a146-207dc12a4e20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Oct 30 05:27:28.404523 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:28.404437    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:27:29.104235 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:29.104196    5898 patch_prober.go:29] interesting pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Oct 30 05:27:29.104638 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:29.104251    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85" podUID="96f0bc26-fbf5-4d4c-a146-207dc12a4e20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Oct 30 05:27:29.251384 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:29.251350    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:27:29.773293 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:29.773255    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:27:29.913549 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:29.913515    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:27:30.104575 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:30.104537    5898 patch_prober.go:29] interesting pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Oct 30 05:27:30.105219 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:30.105180    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85" podUID="96f0bc26-fbf5-4d4c-a146-207dc12a4e20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Oct 30 05:27:30.633875 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:30.633842    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:27:31.104455 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:31.104416    5898 patch_prober.go:29] interesting pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Oct 30 05:27:31.104757 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:31.104728    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85" podUID="96f0bc26-fbf5-4d4c-a146-207dc12a4e20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Oct 30 05:27:32.105306 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:32.105251    5898 patch_prober.go:29] interesting pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Oct 30 05:27:32.105710 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:32.105327    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85" podUID="96f0bc26-fbf5-4d4c-a146-207dc12a4e20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Oct 30 05:27:32.429645 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:32.429563    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:27:32.827070 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:32.826867    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:27:32.967705 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:32.967659    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:27:33.105014 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:33.104872    5898 patch_prober.go:29] interesting pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Oct 30 05:27:33.105014 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:33.105000    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85" podUID="96f0bc26-fbf5-4d4c-a146-207dc12a4e20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Oct 30 05:27:33.534846 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:33.534742    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:27:33.980584 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:33.980535    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:27:34.104205 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:34.104165    5898 patch_prober.go:29] interesting pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Oct 30 05:27:34.104426 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:34.104242    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85" podUID="96f0bc26-fbf5-4d4c-a146-207dc12a4e20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Oct 30 05:27:34.344598 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:34.344565    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:27:34.467244 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:34.467199    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:27:34.644858 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:34.644772    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-93bad9/rate-test-model-93bad9-kserve-6878b4bf49-hcjsm"
Oct 30 05:27:34.663387 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:34.663305    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-93bad9/rate-test-model-93bad9-kserve-6878b4bf49-hcjsm"
Oct 30 05:27:35.013742 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:35.013656    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-93bad9/rate-test-model-93bad9-kserve-6878b4bf49-hcjsm"
Oct 30 05:27:35.111679 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:35.111633    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85"
Oct 30 05:27:35.112313 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:35.112288    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:27:35.112607 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:35.112588    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85"
Oct 30 05:27:35.113438 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:35.113416    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85"
Oct 30 05:27:36.212584 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:36.212547    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:27:36.372161 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:36.372123    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:27:37.226013 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:37.225977    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:27:37.326213 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:37.326177    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:27:38.856356 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:38.856327    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:27:39.017477 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:39.017442    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:27:39.140373 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:39.140292    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:27:40.303394 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:40.303353    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:27:40.831262 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:40.831225    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:27:40.940042 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:40.940013    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:27:42.476834 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:42.476804    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:27:42.603234 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:42.603205    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:27:44.327433 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:44.327397    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:27:46.130875 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:46.130763    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:27:48.047483 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:48.047450    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:27:48.318723 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:48.318631    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:27:49.309758 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:49.309722    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2"]
Oct 30 05:27:49.314958 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:49.314935    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2"
Oct 30 05:27:49.348695 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:49.348661    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2"]
Oct 30 05:27:49.397352 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:49.397312    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb8ee7a6-2a74-411a-8985-c2422f295292-istio-podinfo\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2\" (UID: \"cb8ee7a6-2a74-411a-8985-c2422f295292\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2"
Oct 30 05:27:49.397518 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:49.397405    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb8ee7a6-2a74-411a-8985-c2422f295292-workload-socket\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2\" (UID: \"cb8ee7a6-2a74-411a-8985-c2422f295292\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2"
Oct 30 05:27:49.397518 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:49.397473    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb8ee7a6-2a74-411a-8985-c2422f295292-istio-envoy\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2\" (UID: \"cb8ee7a6-2a74-411a-8985-c2422f295292\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2"
Oct 30 05:27:49.397601 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:49.397530    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb8ee7a6-2a74-411a-8985-c2422f295292-istiod-ca-cert\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2\" (UID: \"cb8ee7a6-2a74-411a-8985-c2422f295292\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2"
Oct 30 05:27:49.397698 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:49.397679    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnpcd\" (UniqueName: \"kubernetes.io/projected/cb8ee7a6-2a74-411a-8985-c2422f295292-kube-api-access-fnpcd\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2\" (UID: \"cb8ee7a6-2a74-411a-8985-c2422f295292\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2"
Oct 30 05:27:49.397762 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:49.397747    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb8ee7a6-2a74-411a-8985-c2422f295292-istio-token\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2\" (UID: \"cb8ee7a6-2a74-411a-8985-c2422f295292\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2"
Oct 30 05:27:49.397803 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:49.397785    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb8ee7a6-2a74-411a-8985-c2422f295292-istio-data\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2\" (UID: \"cb8ee7a6-2a74-411a-8985-c2422f295292\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2"
Oct 30 05:27:49.397916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:49.397887    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb8ee7a6-2a74-411a-8985-c2422f295292-credential-socket\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2\" (UID: \"cb8ee7a6-2a74-411a-8985-c2422f295292\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2"
Oct 30 05:27:49.397963 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:49.397932    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb8ee7a6-2a74-411a-8985-c2422f295292-workload-certs\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2\" (UID: \"cb8ee7a6-2a74-411a-8985-c2422f295292\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2"
Oct 30 05:27:49.499785 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:49.499748    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb8ee7a6-2a74-411a-8985-c2422f295292-credential-socket\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2\" (UID: \"cb8ee7a6-2a74-411a-8985-c2422f295292\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2"
Oct 30 05:27:49.499972 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:49.499796    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb8ee7a6-2a74-411a-8985-c2422f295292-workload-certs\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2\" (UID: \"cb8ee7a6-2a74-411a-8985-c2422f295292\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2"
Oct 30 05:27:49.499972 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:49.499814    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb8ee7a6-2a74-411a-8985-c2422f295292-istio-podinfo\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2\" (UID: \"cb8ee7a6-2a74-411a-8985-c2422f295292\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2"
Oct 30 05:27:49.499972 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:49.499863    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb8ee7a6-2a74-411a-8985-c2422f295292-workload-socket\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2\" (UID: \"cb8ee7a6-2a74-411a-8985-c2422f295292\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2"
Oct 30 05:27:49.499972 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:49.499960    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb8ee7a6-2a74-411a-8985-c2422f295292-istio-envoy\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2\" (UID: \"cb8ee7a6-2a74-411a-8985-c2422f295292\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2"
Oct 30 05:27:49.500185 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:49.500014    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb8ee7a6-2a74-411a-8985-c2422f295292-istiod-ca-cert\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2\" (UID: \"cb8ee7a6-2a74-411a-8985-c2422f295292\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2"
Oct 30 05:27:49.500185 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:49.500143    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnpcd\" (UniqueName: \"kubernetes.io/projected/cb8ee7a6-2a74-411a-8985-c2422f295292-kube-api-access-fnpcd\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2\" (UID: \"cb8ee7a6-2a74-411a-8985-c2422f295292\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2"
Oct 30 05:27:49.500257 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:49.500202    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb8ee7a6-2a74-411a-8985-c2422f295292-istio-token\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2\" (UID: \"cb8ee7a6-2a74-411a-8985-c2422f295292\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2"
Oct 30 05:27:49.500257 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:49.500230    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb8ee7a6-2a74-411a-8985-c2422f295292-istio-data\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2\" (UID: \"cb8ee7a6-2a74-411a-8985-c2422f295292\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2"
Oct 30 05:27:49.500381 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:49.500275    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb8ee7a6-2a74-411a-8985-c2422f295292-workload-certs\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2\" (UID: \"cb8ee7a6-2a74-411a-8985-c2422f295292\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2"
Oct 30 05:27:49.500381 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:49.500277    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb8ee7a6-2a74-411a-8985-c2422f295292-credential-socket\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2\" (UID: \"cb8ee7a6-2a74-411a-8985-c2422f295292\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2"
Oct 30 05:27:49.500877 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:49.500847    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb8ee7a6-2a74-411a-8985-c2422f295292-istiod-ca-cert\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2\" (UID: \"cb8ee7a6-2a74-411a-8985-c2422f295292\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2"
Oct 30 05:27:49.501033 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:49.500940    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb8ee7a6-2a74-411a-8985-c2422f295292-workload-socket\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2\" (UID: \"cb8ee7a6-2a74-411a-8985-c2422f295292\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2"
Oct 30 05:27:49.501033 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:49.500973    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb8ee7a6-2a74-411a-8985-c2422f295292-istio-data\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2\" (UID: \"cb8ee7a6-2a74-411a-8985-c2422f295292\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2"
Oct 30 05:27:49.502610 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:49.502586    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb8ee7a6-2a74-411a-8985-c2422f295292-istio-podinfo\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2\" (UID: \"cb8ee7a6-2a74-411a-8985-c2422f295292\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2"
Oct 30 05:27:49.502704 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:49.502659    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb8ee7a6-2a74-411a-8985-c2422f295292-istio-envoy\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2\" (UID: \"cb8ee7a6-2a74-411a-8985-c2422f295292\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2"
Oct 30 05:27:49.519470 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:49.519443    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb8ee7a6-2a74-411a-8985-c2422f295292-istio-token\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2\" (UID: \"cb8ee7a6-2a74-411a-8985-c2422f295292\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2"
Oct 30 05:27:49.519709 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:49.519678    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnpcd\" (UniqueName: \"kubernetes.io/projected/cb8ee7a6-2a74-411a-8985-c2422f295292-kube-api-access-fnpcd\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2\" (UID: \"cb8ee7a6-2a74-411a-8985-c2422f295292\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2"
Oct 30 05:27:49.630311 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:49.630282    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2"
Oct 30 05:27:49.758523 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:49.758468    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:27:49.905567 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:49.905472    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:27:49.962835 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:49.962794    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2"]
Oct 30 05:27:49.967921 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:27:49.967880    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb8ee7a6_2a74_411a_8985_c2422f295292.slice/crio-d902d7b03df409e39fa8b58ce6512521e26485e62ada7e15bcf44f8d45a0131e WatchSource:0}: Error finding container d902d7b03df409e39fa8b58ce6512521e26485e62ada7e15bcf44f8d45a0131e: Status 404 returned error can't find the container with id d902d7b03df409e39fa8b58ce6512521e26485e62ada7e15bcf44f8d45a0131e
Oct 30 05:27:49.971596 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:49.971549    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:27:49.971706 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:49.971633    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:27:49.971706 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:49.971664    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:27:50.180982 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:50.180033    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2" event={"ID":"cb8ee7a6-2a74-411a-8985-c2422f295292","Type":"ContainerStarted","Data":"dab4512284b5d8df32e9e348f6d2614ee855a106c6a3ef4c763b396cc3c1a072"}
Oct 30 05:27:50.180982 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:50.180114    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2" event={"ID":"cb8ee7a6-2a74-411a-8985-c2422f295292","Type":"ContainerStarted","Data":"d902d7b03df409e39fa8b58ce6512521e26485e62ada7e15bcf44f8d45a0131e"}
Oct 30 05:27:50.630418 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:50.630383    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2"
Oct 30 05:27:51.631964 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:51.631890    5898 patch_prober.go:29] interesting pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Oct 30 05:27:51.632369 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:51.631978    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2" podUID="cb8ee7a6-2a74-411a-8985-c2422f295292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Oct 30 05:27:51.834446 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:51.834393    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:27:52.631228 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:52.631193    5898 patch_prober.go:29] interesting pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Oct 30 05:27:52.631448 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:52.631242    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2" podUID="cb8ee7a6-2a74-411a-8985-c2422f295292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Oct 30 05:27:53.631426 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:53.631382    5898 patch_prober.go:29] interesting pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Oct 30 05:27:53.631988 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:53.631434    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2" podUID="cb8ee7a6-2a74-411a-8985-c2422f295292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Oct 30 05:27:54.011737 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:54.011659    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:27:54.631316 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:54.631275    5898 patch_prober.go:29] interesting pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Oct 30 05:27:54.631542 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:54.631339    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2" podUID="cb8ee7a6-2a74-411a-8985-c2422f295292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Oct 30 05:27:55.631213 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:55.631178    5898 patch_prober.go:29] interesting pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Oct 30 05:27:55.631500 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:55.631236    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2" podUID="cb8ee7a6-2a74-411a-8985-c2422f295292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Oct 30 05:27:55.932779 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:55.932695    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:27:56.630912 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:56.630865    5898 patch_prober.go:29] interesting pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Oct 30 05:27:56.631102 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:56.630965    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2" podUID="cb8ee7a6-2a74-411a-8985-c2422f295292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Oct 30 05:27:56.826197 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:27:56.826156    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:27:57.635932 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:57.635877    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2"
Oct 30 05:27:57.636356 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:57.636325    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2"
Oct 30 05:27:57.637004 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:57.636977    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2"
Oct 30 05:27:57.694789 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:57.694741    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2" podStartSLOduration=8.694727245 podStartE2EDuration="8.694727245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:27:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:27:50.231368081 +0000 UTC m=+2599.455187999" watchObservedRunningTime="2026-10-30 05:27:57.694727245 +0000 UTC m=+2606.918547145"
Oct 30 05:27:57.733563 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:27:57.733538    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:28:00.202684 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:00.202647    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:28:01.065352 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:01.065317    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:28:01.620580 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:01.620535    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:28:01.756156 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:01.756122    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:28:02.416584 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:02.416549    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:28:02.771796 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:02.771706    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:28:02.943052 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:02.943010    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:28:04.401926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:04.394625    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt"]
Oct 30 05:28:04.419722 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:04.419690    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt"]
Oct 30 05:28:04.420072 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:04.420054    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt"
Oct 30 05:28:04.841600 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:04.840367    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b649a15a-bda6-4ad5-852d-8c4253f5900f-istio-data\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt\" (UID: \"b649a15a-bda6-4ad5-852d-8c4253f5900f\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt"
Oct 30 05:28:04.841600 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:04.840430    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b649a15a-bda6-4ad5-852d-8c4253f5900f-istio-podinfo\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt\" (UID: \"b649a15a-bda6-4ad5-852d-8c4253f5900f\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt"
Oct 30 05:28:04.841600 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:04.840562    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b649a15a-bda6-4ad5-852d-8c4253f5900f-credential-socket\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt\" (UID: \"b649a15a-bda6-4ad5-852d-8c4253f5900f\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt"
Oct 30 05:28:04.841600 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:04.840690    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b649a15a-bda6-4ad5-852d-8c4253f5900f-istiod-ca-cert\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt\" (UID: \"b649a15a-bda6-4ad5-852d-8c4253f5900f\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt"
Oct 30 05:28:04.841600 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:04.840745    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b649a15a-bda6-4ad5-852d-8c4253f5900f-workload-socket\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt\" (UID: \"b649a15a-bda6-4ad5-852d-8c4253f5900f\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt"
Oct 30 05:28:04.841600 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:04.840835    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b649a15a-bda6-4ad5-852d-8c4253f5900f-istio-envoy\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt\" (UID: \"b649a15a-bda6-4ad5-852d-8c4253f5900f\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt"
Oct 30 05:28:04.841600 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:04.840863    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfwng\" (UniqueName: \"kubernetes.io/projected/b649a15a-bda6-4ad5-852d-8c4253f5900f-kube-api-access-sfwng\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt\" (UID: \"b649a15a-bda6-4ad5-852d-8c4253f5900f\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt"
Oct 30 05:28:04.841600 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:04.840958    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b649a15a-bda6-4ad5-852d-8c4253f5900f-istio-token\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt\" (UID: \"b649a15a-bda6-4ad5-852d-8c4253f5900f\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt"
Oct 30 05:28:04.841600 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:04.841167    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b649a15a-bda6-4ad5-852d-8c4253f5900f-workload-certs\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt\" (UID: \"b649a15a-bda6-4ad5-852d-8c4253f5900f\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt"
Oct 30 05:28:04.942780 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:04.942741    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b649a15a-bda6-4ad5-852d-8c4253f5900f-credential-socket\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt\" (UID: \"b649a15a-bda6-4ad5-852d-8c4253f5900f\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt"
Oct 30 05:28:04.943009 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:04.942852    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b649a15a-bda6-4ad5-852d-8c4253f5900f-istiod-ca-cert\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt\" (UID: \"b649a15a-bda6-4ad5-852d-8c4253f5900f\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt"
Oct 30 05:28:04.943009 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:04.942888    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b649a15a-bda6-4ad5-852d-8c4253f5900f-workload-socket\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt\" (UID: \"b649a15a-bda6-4ad5-852d-8c4253f5900f\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt"
Oct 30 05:28:04.943009 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:04.942989    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b649a15a-bda6-4ad5-852d-8c4253f5900f-istio-envoy\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt\" (UID: \"b649a15a-bda6-4ad5-852d-8c4253f5900f\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt"
Oct 30 05:28:04.943187 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:04.943011    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfwng\" (UniqueName: \"kubernetes.io/projected/b649a15a-bda6-4ad5-852d-8c4253f5900f-kube-api-access-sfwng\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt\" (UID: \"b649a15a-bda6-4ad5-852d-8c4253f5900f\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt"
Oct 30 05:28:04.943187 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:04.943079    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b649a15a-bda6-4ad5-852d-8c4253f5900f-istio-token\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt\" (UID: \"b649a15a-bda6-4ad5-852d-8c4253f5900f\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt"
Oct 30 05:28:04.943187 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:04.943132    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b649a15a-bda6-4ad5-852d-8c4253f5900f-workload-certs\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt\" (UID: \"b649a15a-bda6-4ad5-852d-8c4253f5900f\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt"
Oct 30 05:28:04.943342 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:04.943226    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b649a15a-bda6-4ad5-852d-8c4253f5900f-istio-data\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt\" (UID: \"b649a15a-bda6-4ad5-852d-8c4253f5900f\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt"
Oct 30 05:28:04.943342 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:04.943251    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b649a15a-bda6-4ad5-852d-8c4253f5900f-credential-socket\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt\" (UID: \"b649a15a-bda6-4ad5-852d-8c4253f5900f\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt"
Oct 30 05:28:04.944051 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:04.943259    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b649a15a-bda6-4ad5-852d-8c4253f5900f-istio-podinfo\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt\" (UID: \"b649a15a-bda6-4ad5-852d-8c4253f5900f\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt"
Oct 30 05:28:04.944051 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:04.943523    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b649a15a-bda6-4ad5-852d-8c4253f5900f-workload-socket\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt\" (UID: \"b649a15a-bda6-4ad5-852d-8c4253f5900f\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt"
Oct 30 05:28:04.944051 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:04.943663    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b649a15a-bda6-4ad5-852d-8c4253f5900f-istiod-ca-cert\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt\" (UID: \"b649a15a-bda6-4ad5-852d-8c4253f5900f\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt"
Oct 30 05:28:04.944051 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:04.943915    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b649a15a-bda6-4ad5-852d-8c4253f5900f-workload-certs\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt\" (UID: \"b649a15a-bda6-4ad5-852d-8c4253f5900f\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt"
Oct 30 05:28:04.944051 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:04.944028    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b649a15a-bda6-4ad5-852d-8c4253f5900f-istio-data\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt\" (UID: \"b649a15a-bda6-4ad5-852d-8c4253f5900f\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt"
Oct 30 05:28:04.945711 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:04.945687    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b649a15a-bda6-4ad5-852d-8c4253f5900f-istio-envoy\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt\" (UID: \"b649a15a-bda6-4ad5-852d-8c4253f5900f\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt"
Oct 30 05:28:04.945991 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:04.945973    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b649a15a-bda6-4ad5-852d-8c4253f5900f-istio-podinfo\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt\" (UID: \"b649a15a-bda6-4ad5-852d-8c4253f5900f\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt"
Oct 30 05:28:04.962801 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:04.962771    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b649a15a-bda6-4ad5-852d-8c4253f5900f-istio-token\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt\" (UID: \"b649a15a-bda6-4ad5-852d-8c4253f5900f\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt"
Oct 30 05:28:04.963501 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:04.963474    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfwng\" (UniqueName: \"kubernetes.io/projected/b649a15a-bda6-4ad5-852d-8c4253f5900f-kube-api-access-sfwng\") pod \"e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt\" (UID: \"b649a15a-bda6-4ad5-852d-8c4253f5900f\") " pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt"
Oct 30 05:28:05.144569 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:05.144483    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt"
Oct 30 05:28:05.334936 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:05.334885    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:28:05.473426 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:05.473395    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt"]
Oct 30 05:28:05.475408 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:28:05.475381    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb649a15a_bda6_4ad5_852d_8c4253f5900f.slice/crio-bb9d1ec260c5c4dd80e4202505fd625167b9c308b360e66d556a837bb1dd9280 WatchSource:0}: Error finding container bb9d1ec260c5c4dd80e4202505fd625167b9c308b360e66d556a837bb1dd9280: Status 404 returned error can't find the container with id bb9d1ec260c5c4dd80e4202505fd625167b9c308b360e66d556a837bb1dd9280
Oct 30 05:28:05.478835 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:05.478799    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:28:05.479166 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:05.478870    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:28:05.479166 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:05.478917    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:28:06.430885 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:06.430845    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt" event={"ID":"b649a15a-bda6-4ad5-852d-8c4253f5900f","Type":"ContainerStarted","Data":"4b53150c7c4665de19177d1eeb2da6a31b842ced080e90e96f8b81776a5eb0c8"}
Oct 30 05:28:06.431092 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:06.430916    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt" event={"ID":"b649a15a-bda6-4ad5-852d-8c4253f5900f","Type":"ContainerStarted","Data":"bb9d1ec260c5c4dd80e4202505fd625167b9c308b360e66d556a837bb1dd9280"}
Oct 30 05:28:06.506396 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:06.506329    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt" podStartSLOduration=2.5062811529999998 podStartE2EDuration="2.506281153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:28:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:28:06.495113609 +0000 UTC m=+2615.718933550" watchObservedRunningTime="2026-10-30 05:28:06.506281153 +0000 UTC m=+2615.730101054"
Oct 30 05:28:07.145206 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:07.145167    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt"
Oct 30 05:28:07.147233 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:07.147205    5898 patch_prober.go:29] interesting pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Oct 30 05:28:07.147355 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:07.147258    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt" podUID="b649a15a-bda6-4ad5-852d-8c4253f5900f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Oct 30 05:28:07.312500 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:07.312458    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:28:07.507974 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:07.501456    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:28:08.145569 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:08.145532    5898 patch_prober.go:29] interesting pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Oct 30 05:28:08.145754 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:08.145591    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt" podUID="b649a15a-bda6-4ad5-852d-8c4253f5900f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Oct 30 05:28:09.136229 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:09.136188    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:28:09.145286 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:09.145253    5898 patch_prober.go:29] interesting pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Oct 30 05:28:09.145407 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:09.145304    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt" podUID="b649a15a-bda6-4ad5-852d-8c4253f5900f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Oct 30 05:28:09.292423 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:09.292307    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:28:10.039216 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:10.039184    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:28:10.148303 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:10.148265    5898 patch_prober.go:29] interesting pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Oct 30 05:28:10.148303 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:10.148329    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt" podUID="b649a15a-bda6-4ad5-852d-8c4253f5900f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Oct 30 05:28:11.144977 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:11.144938    5898 patch_prober.go:29] interesting pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Oct 30 05:28:11.145166 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:11.145005    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt" podUID="b649a15a-bda6-4ad5-852d-8c4253f5900f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Oct 30 05:28:11.817874 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:11.817843    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:28:11.916001 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:11.915963    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh"]
Oct 30 05:28:11.922059 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:11.922026    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh"
Oct 30 05:28:11.933999 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:11.933938    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb"]
Oct 30 05:28:11.941797 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:11.941771    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh"]
Oct 30 05:28:11.941987 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:11.941888    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb"
Oct 30 05:28:11.946504 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:11.946449    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb"]
Oct 30 05:28:12.037838 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.037770    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f-credential-socket\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb\" (UID: \"8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb"
Oct 30 05:28:12.037838 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.037807    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4dc6435-93bb-4b45-bdd5-3a80868e15e3-istio-token\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh\" (UID: \"b4dc6435-93bb-4b45-bdd5-3a80868e15e3\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh"
Oct 30 05:28:12.037838 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.037832    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f-istio-data\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb\" (UID: \"8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb"
Oct 30 05:28:12.038123 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.037893    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4dc6435-93bb-4b45-bdd5-3a80868e15e3-istio-podinfo\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh\" (UID: \"b4dc6435-93bb-4b45-bdd5-3a80868e15e3\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh"
Oct 30 05:28:12.038123 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.037980    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmxl4\" (UniqueName: \"kubernetes.io/projected/b4dc6435-93bb-4b45-bdd5-3a80868e15e3-kube-api-access-hmxl4\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh\" (UID: \"b4dc6435-93bb-4b45-bdd5-3a80868e15e3\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh"
Oct 30 05:28:12.038123 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.038104    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f-istio-podinfo\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb\" (UID: \"8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb"
Oct 30 05:28:12.038232 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.038144    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f-workload-certs\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb\" (UID: \"8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb"
Oct 30 05:28:12.038232 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.038223    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9f86\" (UniqueName: \"kubernetes.io/projected/8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f-kube-api-access-m9f86\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb\" (UID: \"8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb"
Oct 30 05:28:12.038307 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.038249    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f-workload-socket\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb\" (UID: \"8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb"
Oct 30 05:28:12.038307 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.038279    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4dc6435-93bb-4b45-bdd5-3a80868e15e3-istiod-ca-cert\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh\" (UID: \"b4dc6435-93bb-4b45-bdd5-3a80868e15e3\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh"
Oct 30 05:28:12.038384 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.038335    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4dc6435-93bb-4b45-bdd5-3a80868e15e3-istio-envoy\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh\" (UID: \"b4dc6435-93bb-4b45-bdd5-3a80868e15e3\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh"
Oct 30 05:28:12.038384 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.038374    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f-istio-envoy\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb\" (UID: \"8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb"
Oct 30 05:28:12.038453 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.038421    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f-istiod-ca-cert\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb\" (UID: \"8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb"
Oct 30 05:28:12.038488 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.038464    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4dc6435-93bb-4b45-bdd5-3a80868e15e3-credential-socket\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh\" (UID: \"b4dc6435-93bb-4b45-bdd5-3a80868e15e3\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh"
Oct 30 05:28:12.038533 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.038496    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4dc6435-93bb-4b45-bdd5-3a80868e15e3-workload-socket\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh\" (UID: \"b4dc6435-93bb-4b45-bdd5-3a80868e15e3\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh"
Oct 30 05:28:12.038598 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.038585    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4dc6435-93bb-4b45-bdd5-3a80868e15e3-istio-data\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh\" (UID: \"b4dc6435-93bb-4b45-bdd5-3a80868e15e3\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh"
Oct 30 05:28:12.038635 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.038611    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4dc6435-93bb-4b45-bdd5-3a80868e15e3-workload-certs\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh\" (UID: \"b4dc6435-93bb-4b45-bdd5-3a80868e15e3\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh"
Oct 30 05:28:12.038694 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.038681    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f-istio-token\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb\" (UID: \"8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb"
Oct 30 05:28:12.140202 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.140170    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f-credential-socket\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb\" (UID: \"8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb"
Oct 30 05:28:12.140382 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.140216    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4dc6435-93bb-4b45-bdd5-3a80868e15e3-istio-token\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh\" (UID: \"b4dc6435-93bb-4b45-bdd5-3a80868e15e3\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh"
Oct 30 05:28:12.140382 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.140270    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f-istio-data\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb\" (UID: \"8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb"
Oct 30 05:28:12.140382 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.140304    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4dc6435-93bb-4b45-bdd5-3a80868e15e3-istio-podinfo\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh\" (UID: \"b4dc6435-93bb-4b45-bdd5-3a80868e15e3\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh"
Oct 30 05:28:12.140382 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.140332    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmxl4\" (UniqueName: \"kubernetes.io/projected/b4dc6435-93bb-4b45-bdd5-3a80868e15e3-kube-api-access-hmxl4\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh\" (UID: \"b4dc6435-93bb-4b45-bdd5-3a80868e15e3\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh"
Oct 30 05:28:12.140637 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.140425    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f-istio-podinfo\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb\" (UID: \"8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb"
Oct 30 05:28:12.140637 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.140461    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f-workload-certs\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb\" (UID: \"8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb"
Oct 30 05:28:12.140637 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.140566    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9f86\" (UniqueName: \"kubernetes.io/projected/8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f-kube-api-access-m9f86\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb\" (UID: \"8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb"
Oct 30 05:28:12.140637 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.140609    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f-workload-socket\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb\" (UID: \"8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb"
Oct 30 05:28:12.140855 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.140683    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4dc6435-93bb-4b45-bdd5-3a80868e15e3-istiod-ca-cert\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh\" (UID: \"b4dc6435-93bb-4b45-bdd5-3a80868e15e3\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh"
Oct 30 05:28:12.140855 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.140717    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4dc6435-93bb-4b45-bdd5-3a80868e15e3-istio-envoy\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh\" (UID: \"b4dc6435-93bb-4b45-bdd5-3a80868e15e3\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh"
Oct 30 05:28:12.140855 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.140754    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f-istio-envoy\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb\" (UID: \"8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb"
Oct 30 05:28:12.140855 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.140788    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f-istiod-ca-cert\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb\" (UID: \"8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb"
Oct 30 05:28:12.141178 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.140858    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4dc6435-93bb-4b45-bdd5-3a80868e15e3-credential-socket\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh\" (UID: \"b4dc6435-93bb-4b45-bdd5-3a80868e15e3\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh"
Oct 30 05:28:12.141178 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.140924    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f-credential-socket\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb\" (UID: \"8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb"
Oct 30 05:28:12.141178 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.140949    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4dc6435-93bb-4b45-bdd5-3a80868e15e3-workload-socket\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh\" (UID: \"b4dc6435-93bb-4b45-bdd5-3a80868e15e3\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh"
Oct 30 05:28:12.141178 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.141004    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4dc6435-93bb-4b45-bdd5-3a80868e15e3-istio-data\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh\" (UID: \"b4dc6435-93bb-4b45-bdd5-3a80868e15e3\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh"
Oct 30 05:28:12.141178 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.141042    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4dc6435-93bb-4b45-bdd5-3a80868e15e3-workload-certs\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh\" (UID: \"b4dc6435-93bb-4b45-bdd5-3a80868e15e3\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh"
Oct 30 05:28:12.141178 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.141091    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f-istio-token\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb\" (UID: \"8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb"
Oct 30 05:28:12.141532 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.141508    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f-workload-certs\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb\" (UID: \"8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb"
Oct 30 05:28:12.141639 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.141557    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4dc6435-93bb-4b45-bdd5-3a80868e15e3-istiod-ca-cert\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh\" (UID: \"b4dc6435-93bb-4b45-bdd5-3a80868e15e3\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh"
Oct 30 05:28:12.141765 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.140750    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f-istio-data\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb\" (UID: \"8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb"
Oct 30 05:28:12.142227 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.142203    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f-workload-socket\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb\" (UID: \"8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb"
Oct 30 05:28:12.142602 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.142572    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f-istiod-ca-cert\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb\" (UID: \"8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb"
Oct 30 05:28:12.142762 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.142739    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4dc6435-93bb-4b45-bdd5-3a80868e15e3-workload-certs\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh\" (UID: \"b4dc6435-93bb-4b45-bdd5-3a80868e15e3\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh"
Oct 30 05:28:12.143069 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.142955    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4dc6435-93bb-4b45-bdd5-3a80868e15e3-credential-socket\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh\" (UID: \"b4dc6435-93bb-4b45-bdd5-3a80868e15e3\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh"
Oct 30 05:28:12.143411 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.143383    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4dc6435-93bb-4b45-bdd5-3a80868e15e3-workload-socket\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh\" (UID: \"b4dc6435-93bb-4b45-bdd5-3a80868e15e3\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh"
Oct 30 05:28:12.143533 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.143404    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4dc6435-93bb-4b45-bdd5-3a80868e15e3-istio-data\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh\" (UID: \"b4dc6435-93bb-4b45-bdd5-3a80868e15e3\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh"
Oct 30 05:28:12.143793 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.143770    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4dc6435-93bb-4b45-bdd5-3a80868e15e3-istio-podinfo\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh\" (UID: \"b4dc6435-93bb-4b45-bdd5-3a80868e15e3\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh"
Oct 30 05:28:12.144107 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.144084    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4dc6435-93bb-4b45-bdd5-3a80868e15e3-istio-envoy\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh\" (UID: \"b4dc6435-93bb-4b45-bdd5-3a80868e15e3\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh"
Oct 30 05:28:12.144304 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.144282    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f-istio-envoy\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb\" (UID: \"8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb"
Oct 30 05:28:12.144586 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.144564    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f-istio-podinfo\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb\" (UID: \"8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb"
Oct 30 05:28:12.150087 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.150065    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt"
Oct 30 05:28:12.150406 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.150382    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt"
Oct 30 05:28:12.155639 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.155620    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt"
Oct 30 05:28:12.161239 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.161212    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4dc6435-93bb-4b45-bdd5-3a80868e15e3-istio-token\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh\" (UID: \"b4dc6435-93bb-4b45-bdd5-3a80868e15e3\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh"
Oct 30 05:28:12.162575 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.162552    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f-istio-token\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb\" (UID: \"8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb"
Oct 30 05:28:12.164589 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.164564    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmxl4\" (UniqueName: \"kubernetes.io/projected/b4dc6435-93bb-4b45-bdd5-3a80868e15e3-kube-api-access-hmxl4\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh\" (UID: \"b4dc6435-93bb-4b45-bdd5-3a80868e15e3\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh"
Oct 30 05:28:12.165714 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.165690    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9f86\" (UniqueName: \"kubernetes.io/projected/8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f-kube-api-access-m9f86\") pod \"e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb\" (UID: \"8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f\") " pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb"
Oct 30 05:28:12.270947 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.268214    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh"
Oct 30 05:28:12.270947 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.268806    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb"
Oct 30 05:28:12.603200 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.603169    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh"]
Oct 30 05:28:12.605566 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:28:12.605536    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4dc6435_93bb_4b45_bdd5_3a80868e15e3.slice/crio-2c645020fdf4b79ffeaa80e9eb4a45f0b4d83795ef8bed32c42f8a4c487c6d3b WatchSource:0}: Error finding container 2c645020fdf4b79ffeaa80e9eb4a45f0b4d83795ef8bed32c42f8a4c487c6d3b: Status 404 returned error can't find the container with id 2c645020fdf4b79ffeaa80e9eb4a45f0b4d83795ef8bed32c42f8a4c487c6d3b
Oct 30 05:28:12.608278 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.608239    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:28:12.608369 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.608330    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:28:12.608369 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.608359    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:28:12.618830 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.618807    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb"]
Oct 30 05:28:12.622270 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:28:12.622241    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b3a9400_d8c6_4d4d_8be3_92cb66a4bb9f.slice/crio-48bcaf55176641eb56e74bd0f68811fca0c92c1e9e04c44d1ef10666cb858a97 WatchSource:0}: Error finding container 48bcaf55176641eb56e74bd0f68811fca0c92c1e9e04c44d1ef10666cb858a97: Status 404 returned error can't find the container with id 48bcaf55176641eb56e74bd0f68811fca0c92c1e9e04c44d1ef10666cb858a97
Oct 30 05:28:12.625273 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.625243    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:28:12.625369 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.625315    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:28:12.625369 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:12.625363    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:28:13.536294 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:13.536255    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb" event={"ID":"8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f","Type":"ContainerStarted","Data":"f41fe135137090ccc037d32facef08d38129a7fcef77289a60839c9df7d540fb"}
Oct 30 05:28:13.536754 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:13.536324    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb" event={"ID":"8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f","Type":"ContainerStarted","Data":"48bcaf55176641eb56e74bd0f68811fca0c92c1e9e04c44d1ef10666cb858a97"}
Oct 30 05:28:13.538212 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:13.538182    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh" event={"ID":"b4dc6435-93bb-4b45-bdd5-3a80868e15e3","Type":"ContainerStarted","Data":"98c683bf5f41dcf1b722bc2d41507fa4f24694bebd68dcfa7c7957a9a7891dfd"}
Oct 30 05:28:13.538297 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:13.538236    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh" event={"ID":"b4dc6435-93bb-4b45-bdd5-3a80868e15e3","Type":"ContainerStarted","Data":"2c645020fdf4b79ffeaa80e9eb4a45f0b4d83795ef8bed32c42f8a4c487c6d3b"}
Oct 30 05:28:13.661757 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:13.661703    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh" podStartSLOduration=2.66168798 podStartE2EDuration="2.66168798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:28:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:28:13.65922511 +0000 UTC m=+2622.883045029" watchObservedRunningTime="2026-10-30 05:28:13.66168798 +0000 UTC m=+2622.885507880"
Oct 30 05:28:13.662815 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:13.662779    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb" podStartSLOduration=2.662772154 podStartE2EDuration="2.662772154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:28:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:28:13.618303119 +0000 UTC m=+2622.842123040" watchObservedRunningTime="2026-10-30 05:28:13.662772154 +0000 UTC m=+2622.886592057"
Oct 30 05:28:13.746516 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:13.746453    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:28:14.269065 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:14.269032    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh"
Oct 30 05:28:14.270121 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:14.270098    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb"
Oct 30 05:28:14.270846 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:14.270821    5898 patch_prober.go:29] interesting pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 30 05:28:14.270936 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:14.270867    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh" podUID="b4dc6435-93bb-4b45-bdd5-3a80868e15e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 30 05:28:14.271334 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:14.271310    5898 patch_prober.go:29] interesting pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 30 05:28:14.271434 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:14.271356    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb" podUID="8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 30 05:28:15.269310 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:15.269270    5898 patch_prober.go:29] interesting pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 30 05:28:15.269793 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:15.269325    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh" podUID="b4dc6435-93bb-4b45-bdd5-3a80868e15e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 30 05:28:15.269793 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:15.269395    5898 patch_prober.go:29] interesting pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 30 05:28:15.269793 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:15.269421    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb" podUID="8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 30 05:28:15.652921 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:15.650036    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:28:16.183950 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:16.183299    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:28:16.269389 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:16.269351    5898 patch_prober.go:29] interesting pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 30 05:28:16.269944 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:16.269426    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh" podUID="b4dc6435-93bb-4b45-bdd5-3a80868e15e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 30 05:28:16.269944 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:16.269514    5898 patch_prober.go:29] interesting pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 30 05:28:16.269944 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:16.269578    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb" podUID="8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 30 05:28:17.268975 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:17.268945    5898 patch_prober.go:29] interesting pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 30 05:28:17.269239 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:17.269208    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh" podUID="b4dc6435-93bb-4b45-bdd5-3a80868e15e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 30 05:28:17.269476 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:17.269454    5898 patch_prober.go:29] interesting pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 30 05:28:17.269949 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:17.269924    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb" podUID="8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 30 05:28:17.513925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:17.506015    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:28:18.269380 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:18.269346    5898 patch_prober.go:29] interesting pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 30 05:28:18.269570 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:18.269407    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh" podUID="b4dc6435-93bb-4b45-bdd5-3a80868e15e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 30 05:28:18.269570 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:18.269538    5898 patch_prober.go:29] interesting pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 30 05:28:18.269932 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:18.269579    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb" podUID="8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 30 05:28:18.849697 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:18.849650    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww"]
Oct 30 05:28:18.849961 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:18.849861    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww" podUID="d078150b-ed4c-486a-8a9c-762474c26003" containerName="istio-proxy" containerID="cri-o://fa981ab2b4ca444e4733ef9985c1bbeccbbd97c8c8450d3c02cf9d443f154c13" gracePeriod=30
Oct 30 05:28:19.275059 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:19.274982    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh"
Oct 30 05:28:19.275529 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:19.275062    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb"
Oct 30 05:28:19.275529 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:19.275385    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb"
Oct 30 05:28:19.275654 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:19.275582    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh"
Oct 30 05:28:19.276219 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:19.276198    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb"
Oct 30 05:28:19.276469 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:19.276453    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh"
Oct 30 05:28:19.291798 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:19.291770    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:28:21.307282 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:21.307249    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:28:23.156321 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:23.156288    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:28:24.134758 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.134736    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww"
Oct 30 05:28:24.226427 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.226386    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d078150b-ed4c-486a-8a9c-762474c26003-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d078150b-ed4c-486a-8a9c-762474c26003-istio-data\") pod \"d078150b-ed4c-486a-8a9c-762474c26003\" (UID: \"d078150b-ed4c-486a-8a9c-762474c26003\") "
Oct 30 05:28:24.226969 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.226479    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d078150b-ed4c-486a-8a9c-762474c26003-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d078150b-ed4c-486a-8a9c-762474c26003-workload-socket\") pod \"d078150b-ed4c-486a-8a9c-762474c26003\" (UID: \"d078150b-ed4c-486a-8a9c-762474c26003\") "
Oct 30 05:28:24.226969 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.226614    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d078150b-ed4c-486a-8a9c-762474c26003-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d078150b-ed4c-486a-8a9c-762474c26003-credential-socket\") pod \"d078150b-ed4c-486a-8a9c-762474c26003\" (UID: \"d078150b-ed4c-486a-8a9c-762474c26003\") "
Oct 30 05:28:24.226969 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.226671    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d078150b-ed4c-486a-8a9c-762474c26003-kube-api-access-78bxj\" (UniqueName: \"kubernetes.io/projected/d078150b-ed4c-486a-8a9c-762474c26003-kube-api-access-78bxj\") pod \"d078150b-ed4c-486a-8a9c-762474c26003\" (UID: \"d078150b-ed4c-486a-8a9c-762474c26003\") "
Oct 30 05:28:24.226969 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.226712    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d078150b-ed4c-486a-8a9c-762474c26003-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d078150b-ed4c-486a-8a9c-762474c26003-istio-envoy\") pod \"d078150b-ed4c-486a-8a9c-762474c26003\" (UID: \"d078150b-ed4c-486a-8a9c-762474c26003\") "
Oct 30 05:28:24.226969 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.226729    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d078150b-ed4c-486a-8a9c-762474c26003-istio-data" pod "d078150b-ed4c-486a-8a9c-762474c26003" (UID: "d078150b-ed4c-486a-8a9c-762474c26003"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:28:24.226969 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.226747    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d078150b-ed4c-486a-8a9c-762474c26003-istio-token\" (UniqueName: \"kubernetes.io/projected/d078150b-ed4c-486a-8a9c-762474c26003-istio-token\") pod \"d078150b-ed4c-486a-8a9c-762474c26003\" (UID: \"d078150b-ed4c-486a-8a9c-762474c26003\") "
Oct 30 05:28:24.226969 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.226835    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d078150b-ed4c-486a-8a9c-762474c26003-workload-socket" pod "d078150b-ed4c-486a-8a9c-762474c26003" (UID: "d078150b-ed4c-486a-8a9c-762474c26003"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:28:24.226969 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.226842    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d078150b-ed4c-486a-8a9c-762474c26003-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d078150b-ed4c-486a-8a9c-762474c26003-istio-podinfo\") pod \"d078150b-ed4c-486a-8a9c-762474c26003\" (UID: \"d078150b-ed4c-486a-8a9c-762474c26003\") "
Oct 30 05:28:24.226969 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.226967    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d078150b-ed4c-486a-8a9c-762474c26003-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d078150b-ed4c-486a-8a9c-762474c26003-workload-certs\") pod \"d078150b-ed4c-486a-8a9c-762474c26003\" (UID: \"d078150b-ed4c-486a-8a9c-762474c26003\") "
Oct 30 05:28:24.227489 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.227105    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d078150b-ed4c-486a-8a9c-762474c26003-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d078150b-ed4c-486a-8a9c-762474c26003-istiod-ca-cert\") pod \"d078150b-ed4c-486a-8a9c-762474c26003\" (UID: \"d078150b-ed4c-486a-8a9c-762474c26003\") "
Oct 30 05:28:24.228255 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.227999    5898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d078150b-ed4c-486a-8a9c-762474c26003-istio-data\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:28:24.228255 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.228027    5898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d078150b-ed4c-486a-8a9c-762474c26003-workload-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:28:24.228644 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.228612    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d078150b-ed4c-486a-8a9c-762474c26003-istiod-ca-cert" pod "d078150b-ed4c-486a-8a9c-762474c26003" (UID: "d078150b-ed4c-486a-8a9c-762474c26003"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:28:24.228934 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.228874    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d078150b-ed4c-486a-8a9c-762474c26003-workload-certs" pod "d078150b-ed4c-486a-8a9c-762474c26003" (UID: "d078150b-ed4c-486a-8a9c-762474c26003"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:28:24.230308 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.229522    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d078150b-ed4c-486a-8a9c-762474c26003-credential-socket" pod "d078150b-ed4c-486a-8a9c-762474c26003" (UID: "d078150b-ed4c-486a-8a9c-762474c26003"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:28:24.232390 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.232365    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d078150b-ed4c-486a-8a9c-762474c26003-istio-token" pod "d078150b-ed4c-486a-8a9c-762474c26003" (UID: "d078150b-ed4c-486a-8a9c-762474c26003"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:28:24.232496 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.232402    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d078150b-ed4c-486a-8a9c-762474c26003-kube-api-access-78bxj" pod "d078150b-ed4c-486a-8a9c-762474c26003" (UID: "d078150b-ed4c-486a-8a9c-762474c26003"). InnerVolumeSpecName "kube-api-access-78bxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:28:24.232564 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.232502    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d078150b-ed4c-486a-8a9c-762474c26003-istio-podinfo" pod "d078150b-ed4c-486a-8a9c-762474c26003" (UID: "d078150b-ed4c-486a-8a9c-762474c26003"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:28:24.232564 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.232522    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d078150b-ed4c-486a-8a9c-762474c26003-istio-envoy" pod "d078150b-ed4c-486a-8a9c-762474c26003" (UID: "d078150b-ed4c-486a-8a9c-762474c26003"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:28:24.329225 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.329192    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78bxj\" (UniqueName: \"kubernetes.io/projected/d078150b-ed4c-486a-8a9c-762474c26003-kube-api-access-78bxj\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:28:24.329225 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.329223    5898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d078150b-ed4c-486a-8a9c-762474c26003-istio-envoy\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:28:24.329225 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.329233    5898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d078150b-ed4c-486a-8a9c-762474c26003-istio-token\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:28:24.329480 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.329241    5898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d078150b-ed4c-486a-8a9c-762474c26003-istio-podinfo\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:28:24.329480 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.329249    5898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d078150b-ed4c-486a-8a9c-762474c26003-workload-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:28:24.329480 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.329257    5898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d078150b-ed4c-486a-8a9c-762474c26003-istiod-ca-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:28:24.329480 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.329266    5898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d078150b-ed4c-486a-8a9c-762474c26003-credential-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:28:24.516461 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.516373    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss"]
Oct 30 05:28:24.517040 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.517024    5898 cpu_manager.go:414] "containerMap: removing container" podUID="d078150b-ed4c-486a-8a9c-762474c26003" containerName="istio-proxy"
Oct 30 05:28:24.517040 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.517040    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d078150b-ed4c-486a-8a9c-762474c26003" containerName="istio-proxy"
Oct 30 05:28:24.517180 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.517171    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="d078150b-ed4c-486a-8a9c-762474c26003" containerName="istio-proxy"
Oct 30 05:28:24.521195 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.521171    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss"
Oct 30 05:28:24.543219 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.543189    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss"]
Oct 30 05:28:24.632936 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.632889    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e630691-ed49-45ba-ac72-67ef295e80f3-istio-podinfo\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss\" (UID: \"5e630691-ed49-45ba-ac72-67ef295e80f3\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss"
Oct 30 05:28:24.633093 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.632985    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-829pt\" (UniqueName: \"kubernetes.io/projected/5e630691-ed49-45ba-ac72-67ef295e80f3-kube-api-access-829pt\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss\" (UID: \"5e630691-ed49-45ba-ac72-67ef295e80f3\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss"
Oct 30 05:28:24.633093 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.633051    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e630691-ed49-45ba-ac72-67ef295e80f3-istio-data\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss\" (UID: \"5e630691-ed49-45ba-ac72-67ef295e80f3\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss"
Oct 30 05:28:24.633174 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.633090    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e630691-ed49-45ba-ac72-67ef295e80f3-istio-token\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss\" (UID: \"5e630691-ed49-45ba-ac72-67ef295e80f3\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss"
Oct 30 05:28:24.633174 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.633150    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e630691-ed49-45ba-ac72-67ef295e80f3-workload-certs\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss\" (UID: \"5e630691-ed49-45ba-ac72-67ef295e80f3\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss"
Oct 30 05:28:24.633286 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.633270    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e630691-ed49-45ba-ac72-67ef295e80f3-istiod-ca-cert\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss\" (UID: \"5e630691-ed49-45ba-ac72-67ef295e80f3\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss"
Oct 30 05:28:24.633385 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.633373    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e630691-ed49-45ba-ac72-67ef295e80f3-credential-socket\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss\" (UID: \"5e630691-ed49-45ba-ac72-67ef295e80f3\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss"
Oct 30 05:28:24.633468 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.633406    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e630691-ed49-45ba-ac72-67ef295e80f3-istio-envoy\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss\" (UID: \"5e630691-ed49-45ba-ac72-67ef295e80f3\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss"
Oct 30 05:28:24.633468 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.633460    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e630691-ed49-45ba-ac72-67ef295e80f3-workload-socket\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss\" (UID: \"5e630691-ed49-45ba-ac72-67ef295e80f3\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss"
Oct 30 05:28:24.672727 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.672693    5898 generic.go:356] "Generic (PLEG): container finished" podID="d078150b-ed4c-486a-8a9c-762474c26003" containerID="fa981ab2b4ca444e4733ef9985c1bbeccbbd97c8c8450d3c02cf9d443f154c13" exitCode=0
Oct 30 05:28:24.672879 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.672737    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww" event={"ID":"d078150b-ed4c-486a-8a9c-762474c26003","Type":"ContainerDied","Data":"fa981ab2b4ca444e4733ef9985c1bbeccbbd97c8c8450d3c02cf9d443f154c13"}
Oct 30 05:28:24.672879 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.672745    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww"
Oct 30 05:28:24.672879 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.672778    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww" event={"ID":"d078150b-ed4c-486a-8a9c-762474c26003","Type":"ContainerDied","Data":"06b37ff3ec42863fa2748e820eb7f0fbcc7f397bdae8e9d2f66679d2dbfca397"}
Oct 30 05:28:24.672879 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.672801    5898 scope.go:122] "RemoveContainer" containerID="fa981ab2b4ca444e4733ef9985c1bbeccbbd97c8c8450d3c02cf9d443f154c13"
Oct 30 05:28:24.684987 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.684966    5898 scope.go:122] "RemoveContainer" containerID="fa981ab2b4ca444e4733ef9985c1bbeccbbd97c8c8450d3c02cf9d443f154c13"
Oct 30 05:28:24.685292 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:28:24.685268    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa981ab2b4ca444e4733ef9985c1bbeccbbd97c8c8450d3c02cf9d443f154c13\": container with ID starting with fa981ab2b4ca444e4733ef9985c1bbeccbbd97c8c8450d3c02cf9d443f154c13 not found: ID does not exist" containerID="fa981ab2b4ca444e4733ef9985c1bbeccbbd97c8c8450d3c02cf9d443f154c13"
Oct 30 05:28:24.685382 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.685306    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa981ab2b4ca444e4733ef9985c1bbeccbbd97c8c8450d3c02cf9d443f154c13"} err="failed to get container status \"fa981ab2b4ca444e4733ef9985c1bbeccbbd97c8c8450d3c02cf9d443f154c13\": rpc error: code = NotFound desc = could not find container \"fa981ab2b4ca444e4733ef9985c1bbeccbbd97c8c8450d3c02cf9d443f154c13\": container with ID starting with fa981ab2b4ca444e4733ef9985c1bbeccbbd97c8c8450d3c02cf9d443f154c13 not found: ID does not exist"
Oct 30 05:28:24.724486 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.724451    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww"]
Oct 30 05:28:24.732573 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.732547    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-ts8ww"]
Oct 30 05:28:24.734943 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.734916    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e630691-ed49-45ba-ac72-67ef295e80f3-istiod-ca-cert\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss\" (UID: \"5e630691-ed49-45ba-ac72-67ef295e80f3\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss"
Oct 30 05:28:24.735048 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.735032    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e630691-ed49-45ba-ac72-67ef295e80f3-credential-socket\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss\" (UID: \"5e630691-ed49-45ba-ac72-67ef295e80f3\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss"
Oct 30 05:28:24.735094 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.735068    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e630691-ed49-45ba-ac72-67ef295e80f3-istio-envoy\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss\" (UID: \"5e630691-ed49-45ba-ac72-67ef295e80f3\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss"
Oct 30 05:28:24.735155 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.735142    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e630691-ed49-45ba-ac72-67ef295e80f3-workload-socket\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss\" (UID: \"5e630691-ed49-45ba-ac72-67ef295e80f3\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss"
Oct 30 05:28:24.735226 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.735178    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e630691-ed49-45ba-ac72-67ef295e80f3-istio-podinfo\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss\" (UID: \"5e630691-ed49-45ba-ac72-67ef295e80f3\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss"
Oct 30 05:28:24.735226 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.735219    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-829pt\" (UniqueName: \"kubernetes.io/projected/5e630691-ed49-45ba-ac72-67ef295e80f3-kube-api-access-829pt\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss\" (UID: \"5e630691-ed49-45ba-ac72-67ef295e80f3\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss"
Oct 30 05:28:24.735339 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.735261    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e630691-ed49-45ba-ac72-67ef295e80f3-istio-data\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss\" (UID: \"5e630691-ed49-45ba-ac72-67ef295e80f3\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss"
Oct 30 05:28:24.735339 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.735306    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e630691-ed49-45ba-ac72-67ef295e80f3-istio-token\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss\" (UID: \"5e630691-ed49-45ba-ac72-67ef295e80f3\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss"
Oct 30 05:28:24.735445 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.735356    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e630691-ed49-45ba-ac72-67ef295e80f3-workload-certs\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss\" (UID: \"5e630691-ed49-45ba-ac72-67ef295e80f3\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss"
Oct 30 05:28:24.735509 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.735483    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e630691-ed49-45ba-ac72-67ef295e80f3-credential-socket\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss\" (UID: \"5e630691-ed49-45ba-ac72-67ef295e80f3\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss"
Oct 30 05:28:24.735560 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.735527    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e630691-ed49-45ba-ac72-67ef295e80f3-workload-socket\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss\" (UID: \"5e630691-ed49-45ba-ac72-67ef295e80f3\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss"
Oct 30 05:28:24.735708 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.735682    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e630691-ed49-45ba-ac72-67ef295e80f3-istiod-ca-cert\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss\" (UID: \"5e630691-ed49-45ba-ac72-67ef295e80f3\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss"
Oct 30 05:28:24.735816 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.735769    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e630691-ed49-45ba-ac72-67ef295e80f3-istio-data\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss\" (UID: \"5e630691-ed49-45ba-ac72-67ef295e80f3\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss"
Oct 30 05:28:24.736067 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.736046    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e630691-ed49-45ba-ac72-67ef295e80f3-workload-certs\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss\" (UID: \"5e630691-ed49-45ba-ac72-67ef295e80f3\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss"
Oct 30 05:28:24.737535 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.737507    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e630691-ed49-45ba-ac72-67ef295e80f3-istio-envoy\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss\" (UID: \"5e630691-ed49-45ba-ac72-67ef295e80f3\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss"
Oct 30 05:28:24.737755 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.737738    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e630691-ed49-45ba-ac72-67ef295e80f3-istio-podinfo\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss\" (UID: \"5e630691-ed49-45ba-ac72-67ef295e80f3\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss"
Oct 30 05:28:24.755471 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.755448    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e630691-ed49-45ba-ac72-67ef295e80f3-istio-token\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss\" (UID: \"5e630691-ed49-45ba-ac72-67ef295e80f3\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss"
Oct 30 05:28:24.755772 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.755749    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-829pt\" (UniqueName: \"kubernetes.io/projected/5e630691-ed49-45ba-ac72-67ef295e80f3-kube-api-access-829pt\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss\" (UID: \"5e630691-ed49-45ba-ac72-67ef295e80f3\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss"
Oct 30 05:28:24.836570 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.836545    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss"
Oct 30 05:28:24.894542 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.894505    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d078150b-ed4c-486a-8a9c-762474c26003" path="/var/lib/kubelet/pods/d078150b-ed4c-486a-8a9c-762474c26003/volumes"
Oct 30 05:28:24.981547 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:24.981509    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:28:25.088319 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:25.088195    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:28:25.178734 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:25.174430    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss"]
Oct 30 05:28:25.201451 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:25.201363    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:28:25.201640 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:25.201497    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:28:25.201640 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:25.201562    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:28:25.684009 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:25.683974    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss" event={"ID":"5e630691-ed49-45ba-ac72-67ef295e80f3","Type":"ContainerStarted","Data":"cd08ba50e7496e9421154554582d82cb6cbcbe93039b6b211b09d4405eb5be86"}
Oct 30 05:28:25.684505 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:25.684039    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss" event={"ID":"5e630691-ed49-45ba-ac72-67ef295e80f3","Type":"ContainerStarted","Data":"49348ed659d5f54a3773de72a145cf4510cceb477191b99015e0541d6a0ced1c"}
Oct 30 05:28:25.731624 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:25.731558    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss" podStartSLOduration=1.731539636 podStartE2EDuration="1.731539636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:28:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:28:25.726101663 +0000 UTC m=+2634.949921566" watchObservedRunningTime="2026-10-30 05:28:25.731539636 +0000 UTC m=+2634.955359541"
Oct 30 05:28:25.837073 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:25.837040    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss"
Oct 30 05:28:26.219388 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:26.219353    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-6b3855/ipp-route-6b3855-kserve-f8c8cf4f7-4pnld"]
Oct 30 05:28:26.223954 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:26.223936    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-6b3855/ipp-route-6b3855-kserve-f8c8cf4f7-4pnld"
Oct 30 05:28:26.243113 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:26.243080    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-6b3855/ipp-route-6b3855-kserve-f8c8cf4f7-4pnld"]
Oct 30 05:28:26.249960 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:26.249937    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-6b3855\"/\"default-dockercfg-f4mnn\""
Oct 30 05:28:26.250244 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:26.250228    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-6b3855\"/\"ipp-route-6b3855-kserve-self-signed-certs\""
Oct 30 05:28:26.252863 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:26.252843    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-6b3855\"/\"kube-root-ca.crt\""
Oct 30 05:28:26.254932 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:26.254892    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-6b3855\"/\"openshift-service-ca.crt\""
Oct 30 05:28:26.349373 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:26.349335    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:28:26.409936 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:26.393749    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0af0863a-6896-443d-be5a-bbb5a1e0edae-dshm\") pod \"ipp-route-6b3855-kserve-f8c8cf4f7-4pnld\" (UID: \"0af0863a-6896-443d-be5a-bbb5a1e0edae\") " pod="ai-tenant-e2e-ipp-a-6b3855/ipp-route-6b3855-kserve-f8c8cf4f7-4pnld"
Oct 30 05:28:26.409936 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:26.393875    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0af0863a-6896-443d-be5a-bbb5a1e0edae-home\") pod \"ipp-route-6b3855-kserve-f8c8cf4f7-4pnld\" (UID: \"0af0863a-6896-443d-be5a-bbb5a1e0edae\") " pod="ai-tenant-e2e-ipp-a-6b3855/ipp-route-6b3855-kserve-f8c8cf4f7-4pnld"
Oct 30 05:28:26.409936 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:26.393931    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0af0863a-6896-443d-be5a-bbb5a1e0edae-model-cache\") pod \"ipp-route-6b3855-kserve-f8c8cf4f7-4pnld\" (UID: \"0af0863a-6896-443d-be5a-bbb5a1e0edae\") " pod="ai-tenant-e2e-ipp-a-6b3855/ipp-route-6b3855-kserve-f8c8cf4f7-4pnld"
Oct 30 05:28:26.409936 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:26.393981    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5bhs\" (UniqueName: \"kubernetes.io/projected/0af0863a-6896-443d-be5a-bbb5a1e0edae-kube-api-access-k5bhs\") pod \"ipp-route-6b3855-kserve-f8c8cf4f7-4pnld\" (UID: \"0af0863a-6896-443d-be5a-bbb5a1e0edae\") " pod="ai-tenant-e2e-ipp-a-6b3855/ipp-route-6b3855-kserve-f8c8cf4f7-4pnld"
Oct 30 05:28:26.409936 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:26.394135    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0af0863a-6896-443d-be5a-bbb5a1e0edae-tmp-dir\") pod \"ipp-route-6b3855-kserve-f8c8cf4f7-4pnld\" (UID: \"0af0863a-6896-443d-be5a-bbb5a1e0edae\") " pod="ai-tenant-e2e-ipp-a-6b3855/ipp-route-6b3855-kserve-f8c8cf4f7-4pnld"
Oct 30 05:28:26.409936 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:26.394425    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0af0863a-6896-443d-be5a-bbb5a1e0edae-tls-certs\") pod \"ipp-route-6b3855-kserve-f8c8cf4f7-4pnld\" (UID: \"0af0863a-6896-443d-be5a-bbb5a1e0edae\") " pod="ai-tenant-e2e-ipp-a-6b3855/ipp-route-6b3855-kserve-f8c8cf4f7-4pnld"
Oct 30 05:28:26.499976 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:26.496948    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0af0863a-6896-443d-be5a-bbb5a1e0edae-tls-certs\") pod \"ipp-route-6b3855-kserve-f8c8cf4f7-4pnld\" (UID: \"0af0863a-6896-443d-be5a-bbb5a1e0edae\") " pod="ai-tenant-e2e-ipp-a-6b3855/ipp-route-6b3855-kserve-f8c8cf4f7-4pnld"
Oct 30 05:28:26.499976 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:26.497030    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0af0863a-6896-443d-be5a-bbb5a1e0edae-dshm\") pod \"ipp-route-6b3855-kserve-f8c8cf4f7-4pnld\" (UID: \"0af0863a-6896-443d-be5a-bbb5a1e0edae\") " pod="ai-tenant-e2e-ipp-a-6b3855/ipp-route-6b3855-kserve-f8c8cf4f7-4pnld"
Oct 30 05:28:26.499976 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:26.497083    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0af0863a-6896-443d-be5a-bbb5a1e0edae-home\") pod \"ipp-route-6b3855-kserve-f8c8cf4f7-4pnld\" (UID: \"0af0863a-6896-443d-be5a-bbb5a1e0edae\") " pod="ai-tenant-e2e-ipp-a-6b3855/ipp-route-6b3855-kserve-f8c8cf4f7-4pnld"
Oct 30 05:28:26.499976 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:26.497107    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0af0863a-6896-443d-be5a-bbb5a1e0edae-model-cache\") pod \"ipp-route-6b3855-kserve-f8c8cf4f7-4pnld\" (UID: \"0af0863a-6896-443d-be5a-bbb5a1e0edae\") " pod="ai-tenant-e2e-ipp-a-6b3855/ipp-route-6b3855-kserve-f8c8cf4f7-4pnld"
Oct 30 05:28:26.499976 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:26.497142    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5bhs\" (UniqueName: \"kubernetes.io/projected/0af0863a-6896-443d-be5a-bbb5a1e0edae-kube-api-access-k5bhs\") pod \"ipp-route-6b3855-kserve-f8c8cf4f7-4pnld\" (UID: \"0af0863a-6896-443d-be5a-bbb5a1e0edae\") " pod="ai-tenant-e2e-ipp-a-6b3855/ipp-route-6b3855-kserve-f8c8cf4f7-4pnld"
Oct 30 05:28:26.499976 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:26.497307    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0af0863a-6896-443d-be5a-bbb5a1e0edae-tmp-dir\") pod \"ipp-route-6b3855-kserve-f8c8cf4f7-4pnld\" (UID: \"0af0863a-6896-443d-be5a-bbb5a1e0edae\") " pod="ai-tenant-e2e-ipp-a-6b3855/ipp-route-6b3855-kserve-f8c8cf4f7-4pnld"
Oct 30 05:28:26.499976 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:26.497928    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0af0863a-6896-443d-be5a-bbb5a1e0edae-tmp-dir\") pod \"ipp-route-6b3855-kserve-f8c8cf4f7-4pnld\" (UID: \"0af0863a-6896-443d-be5a-bbb5a1e0edae\") " pod="ai-tenant-e2e-ipp-a-6b3855/ipp-route-6b3855-kserve-f8c8cf4f7-4pnld"
Oct 30 05:28:26.499976 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:26.498592    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0af0863a-6896-443d-be5a-bbb5a1e0edae-home\") pod \"ipp-route-6b3855-kserve-f8c8cf4f7-4pnld\" (UID: \"0af0863a-6896-443d-be5a-bbb5a1e0edae\") " pod="ai-tenant-e2e-ipp-a-6b3855/ipp-route-6b3855-kserve-f8c8cf4f7-4pnld"
Oct 30 05:28:26.499976 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:26.499113    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0af0863a-6896-443d-be5a-bbb5a1e0edae-model-cache\") pod \"ipp-route-6b3855-kserve-f8c8cf4f7-4pnld\" (UID: \"0af0863a-6896-443d-be5a-bbb5a1e0edae\") " pod="ai-tenant-e2e-ipp-a-6b3855/ipp-route-6b3855-kserve-f8c8cf4f7-4pnld"
Oct 30 05:28:26.506000 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:26.505974    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0af0863a-6896-443d-be5a-bbb5a1e0edae-tls-certs\") pod \"ipp-route-6b3855-kserve-f8c8cf4f7-4pnld\" (UID: \"0af0863a-6896-443d-be5a-bbb5a1e0edae\") " pod="ai-tenant-e2e-ipp-a-6b3855/ipp-route-6b3855-kserve-f8c8cf4f7-4pnld"
Oct 30 05:28:26.507242 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:26.507218    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0af0863a-6896-443d-be5a-bbb5a1e0edae-dshm\") pod \"ipp-route-6b3855-kserve-f8c8cf4f7-4pnld\" (UID: \"0af0863a-6896-443d-be5a-bbb5a1e0edae\") " pod="ai-tenant-e2e-ipp-a-6b3855/ipp-route-6b3855-kserve-f8c8cf4f7-4pnld"
Oct 30 05:28:26.527762 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:26.527728    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5bhs\" (UniqueName: \"kubernetes.io/projected/0af0863a-6896-443d-be5a-bbb5a1e0edae-kube-api-access-k5bhs\") pod \"ipp-route-6b3855-kserve-f8c8cf4f7-4pnld\" (UID: \"0af0863a-6896-443d-be5a-bbb5a1e0edae\") " pod="ai-tenant-e2e-ipp-a-6b3855/ipp-route-6b3855-kserve-f8c8cf4f7-4pnld"
Oct 30 05:28:26.543882 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:26.543851    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-6b3855/ipp-route-6b3855-kserve-f8c8cf4f7-4pnld"
Oct 30 05:28:26.849925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:26.843561    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Oct 30 05:28:26.849925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:26.843633    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss" podUID="5e630691-ed49-45ba-ac72-67ef295e80f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Oct 30 05:28:26.973971 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:28:26.973258    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:28:27.017386 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:27.017291    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-6b3855/ipp-route-6b3855-kserve-f8c8cf4f7-4pnld"]
Oct 30 05:28:27.296925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:27.282604    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:28:27.837253 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:27.837217    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Oct 30 05:28:27.837432 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:27.837273    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss" podUID="5e630691-ed49-45ba-ac72-67ef295e80f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Oct 30 05:28:27.911644 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:27.911599    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-6b3855/ipp-route-6b3855-kserve-f8c8cf4f7-4pnld" event={"ID":"0af0863a-6896-443d-be5a-bbb5a1e0edae","Type":"ContainerStarted","Data":"5bcd9474c29a3f760f8ac4a5e3342b3972006392c4b63f0a4cc72e097dc08283"}
Oct 30 05:28:27.912147 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:27.911654    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-6b3855/ipp-route-6b3855-kserve-f8c8cf4f7-4pnld" event={"ID":"0af0863a-6896-443d-be5a-bbb5a1e0edae","Type":"ContainerStarted","Data":"24a8b9aa783dcf86bc7154c111bc588fbe52e40f0dbd766a867342c6422acbee"}
Oct 30 05:28:27.954250 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:27.954199    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-6b3855/ipp-route-6b3855-kserve-f8c8cf4f7-4pnld" podStartSLOduration=1.954184978 podStartE2EDuration="1.954184978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:28:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:28:27.950254942 +0000 UTC m=+2637.174074862" watchObservedRunningTime="2026-10-30 05:28:27.954184978 +0000 UTC m=+2637.178004877"
Oct 30 05:28:28.544470 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:28.544438    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-6b3855/ipp-route-6b3855-kserve-f8c8cf4f7-4pnld"
Oct 30 05:28:28.837876 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:28.837843    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Oct 30 05:28:28.838074 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:28.837916    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss" podUID="5e630691-ed49-45ba-ac72-67ef295e80f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Oct 30 05:28:29.149285 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:29.149192    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:28:29.534272 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:29.534183    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd"]
Oct 30 05:28:29.538741 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:29.538714    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd"
Oct 30 05:28:29.567023 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:29.566117    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-dfb231-openshift-default-dockercfg-t5h89\""
Oct 30 05:28:29.618941 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:29.617844    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd"]
Oct 30 05:28:29.648095 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:29.648054    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4fd7194-084b-4339-9ea3-c5601858fed5-workload-socket\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd\" (UID: \"d4fd7194-084b-4339-9ea3-c5601858fed5\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd"
Oct 30 05:28:29.648296 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:29.648198    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4fd7194-084b-4339-9ea3-c5601858fed5-credential-socket\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd\" (UID: \"d4fd7194-084b-4339-9ea3-c5601858fed5\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd"
Oct 30 05:28:29.648296 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:29.648246    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4fd7194-084b-4339-9ea3-c5601858fed5-workload-certs\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd\" (UID: \"d4fd7194-084b-4339-9ea3-c5601858fed5\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd"
Oct 30 05:28:29.648296 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:29.648288    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4fd7194-084b-4339-9ea3-c5601858fed5-istio-podinfo\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd\" (UID: \"d4fd7194-084b-4339-9ea3-c5601858fed5\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd"
Oct 30 05:28:29.648561 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:29.648356    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4fd7194-084b-4339-9ea3-c5601858fed5-istiod-ca-cert\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd\" (UID: \"d4fd7194-084b-4339-9ea3-c5601858fed5\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd"
Oct 30 05:28:29.648561 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:29.648418    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4fd7194-084b-4339-9ea3-c5601858fed5-istio-envoy\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd\" (UID: \"d4fd7194-084b-4339-9ea3-c5601858fed5\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd"
Oct 30 05:28:29.648561 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:29.648450    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4fd7194-084b-4339-9ea3-c5601858fed5-istio-data\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd\" (UID: \"d4fd7194-084b-4339-9ea3-c5601858fed5\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd"
Oct 30 05:28:29.648561 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:29.648481    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4fd7194-084b-4339-9ea3-c5601858fed5-istio-token\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd\" (UID: \"d4fd7194-084b-4339-9ea3-c5601858fed5\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd"
Oct 30 05:28:29.648561 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:29.648497    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgncq\" (UniqueName: \"kubernetes.io/projected/d4fd7194-084b-4339-9ea3-c5601858fed5-kube-api-access-xgncq\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd\" (UID: \"d4fd7194-084b-4339-9ea3-c5601858fed5\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd"
Oct 30 05:28:29.749807 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:29.749768    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4fd7194-084b-4339-9ea3-c5601858fed5-istiod-ca-cert\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd\" (UID: \"d4fd7194-084b-4339-9ea3-c5601858fed5\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd"
Oct 30 05:28:29.750021 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:29.749871    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4fd7194-084b-4339-9ea3-c5601858fed5-istio-envoy\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd\" (UID: \"d4fd7194-084b-4339-9ea3-c5601858fed5\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd"
Oct 30 05:28:29.750021 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:29.749934    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4fd7194-084b-4339-9ea3-c5601858fed5-istio-data\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd\" (UID: \"d4fd7194-084b-4339-9ea3-c5601858fed5\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd"
Oct 30 05:28:29.750021 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:29.749981    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4fd7194-084b-4339-9ea3-c5601858fed5-istio-token\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd\" (UID: \"d4fd7194-084b-4339-9ea3-c5601858fed5\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd"
Oct 30 05:28:29.750021 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:29.750004    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgncq\" (UniqueName: \"kubernetes.io/projected/d4fd7194-084b-4339-9ea3-c5601858fed5-kube-api-access-xgncq\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd\" (UID: \"d4fd7194-084b-4339-9ea3-c5601858fed5\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd"
Oct 30 05:28:29.750249 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:29.750079    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4fd7194-084b-4339-9ea3-c5601858fed5-workload-socket\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd\" (UID: \"d4fd7194-084b-4339-9ea3-c5601858fed5\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd"
Oct 30 05:28:29.750249 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:29.750185    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4fd7194-084b-4339-9ea3-c5601858fed5-credential-socket\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd\" (UID: \"d4fd7194-084b-4339-9ea3-c5601858fed5\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd"
Oct 30 05:28:29.750354 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:29.750246    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4fd7194-084b-4339-9ea3-c5601858fed5-workload-certs\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd\" (UID: \"d4fd7194-084b-4339-9ea3-c5601858fed5\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd"
Oct 30 05:28:29.750354 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:29.750311    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4fd7194-084b-4339-9ea3-c5601858fed5-istio-podinfo\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd\" (UID: \"d4fd7194-084b-4339-9ea3-c5601858fed5\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd"
Oct 30 05:28:29.750462 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:29.750396    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4fd7194-084b-4339-9ea3-c5601858fed5-istio-data\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd\" (UID: \"d4fd7194-084b-4339-9ea3-c5601858fed5\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd"
Oct 30 05:28:29.750462 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:29.750439    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4fd7194-084b-4339-9ea3-c5601858fed5-workload-socket\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd\" (UID: \"d4fd7194-084b-4339-9ea3-c5601858fed5\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd"
Oct 30 05:28:29.750651 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:29.750630    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4fd7194-084b-4339-9ea3-c5601858fed5-credential-socket\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd\" (UID: \"d4fd7194-084b-4339-9ea3-c5601858fed5\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd"
Oct 30 05:28:29.750731 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:29.750653    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4fd7194-084b-4339-9ea3-c5601858fed5-istiod-ca-cert\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd\" (UID: \"d4fd7194-084b-4339-9ea3-c5601858fed5\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd"
Oct 30 05:28:29.750731 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:29.750672    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4fd7194-084b-4339-9ea3-c5601858fed5-workload-certs\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd\" (UID: \"d4fd7194-084b-4339-9ea3-c5601858fed5\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd"
Oct 30 05:28:29.752679 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:29.752650    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4fd7194-084b-4339-9ea3-c5601858fed5-istio-envoy\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd\" (UID: \"d4fd7194-084b-4339-9ea3-c5601858fed5\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd"
Oct 30 05:28:29.752796 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:29.752718    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4fd7194-084b-4339-9ea3-c5601858fed5-istio-podinfo\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd\" (UID: \"d4fd7194-084b-4339-9ea3-c5601858fed5\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd"
Oct 30 05:28:29.768722 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:29.768701    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4fd7194-084b-4339-9ea3-c5601858fed5-istio-token\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd\" (UID: \"d4fd7194-084b-4339-9ea3-c5601858fed5\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd"
Oct 30 05:28:29.769084 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:29.769062    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgncq\" (UniqueName: \"kubernetes.io/projected/d4fd7194-084b-4339-9ea3-c5601858fed5-kube-api-access-xgncq\") pod \"e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd\" (UID: \"d4fd7194-084b-4339-9ea3-c5601858fed5\") " pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd"
Oct 30 05:28:29.837032 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:29.837004    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Oct 30 05:28:29.837203 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:29.837055    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss" podUID="5e630691-ed49-45ba-ac72-67ef295e80f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Oct 30 05:28:29.860564 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:29.860537    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd"
Oct 30 05:28:30.214875 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:30.214680    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd"]
Oct 30 05:28:30.217710 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:28:30.217679    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4fd7194_084b_4339_9ea3_c5601858fed5.slice/crio-60807dc4fea3e3a2a31b9068ee212a5cd67e18b16f49189a9941ae3b96fcfff8 WatchSource:0}: Error finding container 60807dc4fea3e3a2a31b9068ee212a5cd67e18b16f49189a9941ae3b96fcfff8: Status 404 returned error can't find the container with id 60807dc4fea3e3a2a31b9068ee212a5cd67e18b16f49189a9941ae3b96fcfff8
Oct 30 05:28:30.220780 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:30.220743    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:28:30.220935 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:30.220831    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:28:30.220935 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:30.220877    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:28:30.799922 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:30.792276    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:28:30.837504 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:30.837469    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Oct 30 05:28:30.837691 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:30.837533    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss" podUID="5e630691-ed49-45ba-ac72-67ef295e80f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Oct 30 05:28:30.919593 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:30.919558    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:28:30.936081 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:30.936045    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd" event={"ID":"d4fd7194-084b-4339-9ea3-c5601858fed5","Type":"ContainerStarted","Data":"19841e4f8accb3a6b8f541099d9cbafdc222df3c6d16369aed86662c69c57f58"}
Oct 30 05:28:30.936250 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:30.936104    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd" event={"ID":"d4fd7194-084b-4339-9ea3-c5601858fed5","Type":"ContainerStarted","Data":"60807dc4fea3e3a2a31b9068ee212a5cd67e18b16f49189a9941ae3b96fcfff8"}
Oct 30 05:28:31.014035 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:31.013972    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd" podStartSLOduration=2.013952646 podStartE2EDuration="2.013952646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:28:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:28:30.994506366 +0000 UTC m=+2640.218326319" watchObservedRunningTime="2026-10-30 05:28:31.013952646 +0000 UTC m=+2640.237772560"
Oct 30 05:28:31.621925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:31.613194    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:28:31.779936 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:31.761273    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:28:31.840325 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:31.840287    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Oct 30 05:28:31.840547 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:31.840346    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss" podUID="5e630691-ed49-45ba-ac72-67ef295e80f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Oct 30 05:28:31.860806 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:31.860770    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd"
Oct 30 05:28:31.862496 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:31.862470    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 30 05:28:31.862622 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:31.862523    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd" podUID="d4fd7194-084b-4339-9ea3-c5601858fed5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 30 05:28:31.869047 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:31.869023    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:28:32.927922 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:32.924474    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 30 05:28:32.927922 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:32.924537    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd" podUID="d4fd7194-084b-4339-9ea3-c5601858fed5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 30 05:28:33.839845 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:33.839807    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:28:33.851647 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:33.850994    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 05:28:33.851647 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:33.851061    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss" podUID="5e630691-ed49-45ba-ac72-67ef295e80f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:28:33.866718 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:33.866686    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 30 05:28:33.868440 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:33.868367    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd" podUID="d4fd7194-084b-4339-9ea3-c5601858fed5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 30 05:28:34.139483 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:34.139390    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss"
Oct 30 05:28:34.139951 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:34.139928    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss"
Oct 30 05:28:34.141269 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:34.141240    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss"
Oct 30 05:28:34.861685 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:34.861643    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 30 05:28:34.861866 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:34.861713    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd" podUID="d4fd7194-084b-4339-9ea3-c5601858fed5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 30 05:28:35.550745 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:35.550708    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:28:35.861663 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:35.861622    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 30 05:28:35.861848 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:35.861695    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd" podUID="d4fd7194-084b-4339-9ea3-c5601858fed5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 30 05:28:36.545024 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:36.544991    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-6b3855/ipp-route-6b3855-kserve-f8c8cf4f7-4pnld"
Oct 30 05:28:36.558531 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:36.558507    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-6b3855/ipp-route-6b3855-kserve-f8c8cf4f7-4pnld"
Oct 30 05:28:36.861590 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:36.861549    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 30 05:28:36.861774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:36.861606    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd" podUID="d4fd7194-084b-4339-9ea3-c5601858fed5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 30 05:28:37.142081 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:37.141993    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-6b3855/ipp-route-6b3855-kserve-f8c8cf4f7-4pnld"
Oct 30 05:28:37.400660 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:37.400549    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:28:37.922042 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:37.922007    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd"
Oct 30 05:28:37.922663 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:37.922377    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd"
Oct 30 05:28:37.923382 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:37.923312    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd"
Oct 30 05:28:39.078511 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:39.078480    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:28:40.849768 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:40.849710    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:28:41.348957 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:41.348882    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:28:42.723542 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:42.723503    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:28:42.886832 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:42.886777    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:28:43.197765 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:43.197727    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:28:44.645361 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:44.645306    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:28:46.392367 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:46.390768    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:28:48.090431 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:48.090375    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:28:48.193978 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:48.193887    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:28:49.663991 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:49.663954    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw"]
Oct 30 05:28:49.669276 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:49.669252    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw"
Oct 30 05:28:49.684238 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:49.684213    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-34722e-openshift-default-dockercfg-nq64s\""
Oct 30 05:28:49.700615 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:49.700588    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw"]
Oct 30 05:28:49.750917 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:49.750867    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd-istio-envoy\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw\" (UID: \"34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw"
Oct 30 05:28:49.751091 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:49.750979    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd-istio-podinfo\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw\" (UID: \"34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw"
Oct 30 05:28:49.751091 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:49.751028    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd-istio-data\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw\" (UID: \"34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw"
Oct 30 05:28:49.751186 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:49.751151    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd-istio-token\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw\" (UID: \"34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw"
Oct 30 05:28:49.751229 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:49.751196    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd-workload-socket\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw\" (UID: \"34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw"
Oct 30 05:28:49.751229 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:49.751219    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd-workload-certs\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw\" (UID: \"34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw"
Oct 30 05:28:49.751363 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:49.751348    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxgbk\" (UniqueName: \"kubernetes.io/projected/34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd-kube-api-access-gxgbk\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw\" (UID: \"34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw"
Oct 30 05:28:49.751401 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:49.751391    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd-istiod-ca-cert\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw\" (UID: \"34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw"
Oct 30 05:28:49.751528 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:49.751503    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd-credential-socket\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw\" (UID: \"34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw"
Oct 30 05:28:49.796967 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:49.796940    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:28:49.858235 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:49.858190    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd-istio-envoy\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw\" (UID: \"34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw"
Oct 30 05:28:49.859046 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:49.859020    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd-istio-podinfo\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw\" (UID: \"34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw"
Oct 30 05:28:49.859289 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:49.859250    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd-istio-data\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw\" (UID: \"34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw"
Oct 30 05:28:49.870256 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:49.870220    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd-istio-token\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw\" (UID: \"34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw"
Oct 30 05:28:49.870513 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:49.870489    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd-workload-socket\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw\" (UID: \"34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw"
Oct 30 05:28:49.870681 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:49.870660    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd-workload-certs\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw\" (UID: \"34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw"
Oct 30 05:28:49.884851 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:49.870984    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxgbk\" (UniqueName: \"kubernetes.io/projected/34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd-kube-api-access-gxgbk\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw\" (UID: \"34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw"
Oct 30 05:28:49.885613 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:49.885584    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd-istiod-ca-cert\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw\" (UID: \"34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw"
Oct 30 05:28:49.886662 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:49.872830    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd-istio-data\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw\" (UID: \"34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw"
Oct 30 05:28:49.887029 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:49.873115    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd-workload-certs\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw\" (UID: \"34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw"
Oct 30 05:28:49.887199 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:49.872601    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd-workload-socket\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw\" (UID: \"34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw"
Oct 30 05:28:49.887330 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:49.886504    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd-istiod-ca-cert\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw\" (UID: \"34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw"
Oct 30 05:28:49.887431 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:49.886624    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd-credential-socket\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw\" (UID: \"34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw"
Oct 30 05:28:49.887796 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:49.886861    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd-credential-socket\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw\" (UID: \"34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw"
Oct 30 05:28:49.897737 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:49.897710    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd-istio-podinfo\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw\" (UID: \"34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw"
Oct 30 05:28:49.909581 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:49.909548    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd-istio-envoy\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw\" (UID: \"34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw"
Oct 30 05:28:49.915823 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:49.915758    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxgbk\" (UniqueName: \"kubernetes.io/projected/34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd-kube-api-access-gxgbk\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw\" (UID: \"34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw"
Oct 30 05:28:49.916439 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:49.916417    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd-istio-token\") pod \"e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw\" (UID: \"34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd\") " pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw"
Oct 30 05:28:49.984641 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:49.984609    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw"
Oct 30 05:28:50.332422 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:50.332394    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw"]
Oct 30 05:28:50.334297 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:28:50.334267    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34fc87bb_0916_4cd3_bec8_d3ae0d7af7cd.slice/crio-106b82f84b7f1734e762195e7e00dd4b6246a7eefe7399c8c0f8b7def44e5ef2 WatchSource:0}: Error finding container 106b82f84b7f1734e762195e7e00dd4b6246a7eefe7399c8c0f8b7def44e5ef2: Status 404 returned error can't find the container with id 106b82f84b7f1734e762195e7e00dd4b6246a7eefe7399c8c0f8b7def44e5ef2
Oct 30 05:28:50.337634 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:50.337594    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:28:50.337720 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:50.337679    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:28:50.337770 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:50.337719    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:28:51.289070 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:51.289036    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw" event={"ID":"34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd","Type":"ContainerStarted","Data":"a54c751205acaae25e07f729e2ed2cb3856bc8a87f507a6170a2f74ff6711d44"}
Oct 30 05:28:51.289482 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:51.289094    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw" event={"ID":"34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd","Type":"ContainerStarted","Data":"106b82f84b7f1734e762195e7e00dd4b6246a7eefe7399c8c0f8b7def44e5ef2"}
Oct 30 05:28:51.357361 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:51.357306    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw" podStartSLOduration=2.357292839 podStartE2EDuration="2.357292839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:28:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:28:51.342244822 +0000 UTC m=+2660.566064724" watchObservedRunningTime="2026-10-30 05:28:51.357292839 +0000 UTC m=+2660.581112739"
Oct 30 05:28:51.995521 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:51.995247    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw"
Oct 30 05:28:52.001303 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:51.999076    5898 patch_prober.go:29] interesting pod/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 05:28:52.001303 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:51.999147    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw" podUID="34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 05:28:52.211025 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:52.210989    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:28:52.997384 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:52.997295    5898 patch_prober.go:29] interesting pod/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 05:28:52.997981 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:52.997950    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw" podUID="34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 05:28:53.986355 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:53.986314    5898 patch_prober.go:29] interesting pod/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 05:28:53.986659 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:53.986626    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw" podUID="34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 05:28:54.500362 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:54.500327    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:28:54.993926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:54.989248    5898 patch_prober.go:29] interesting pod/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 05:28:54.993926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:54.989309    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw" podUID="34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 05:28:55.985294 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:55.985257    5898 patch_prober.go:29] interesting pod/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 05:28:55.987020 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:55.986989    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw" podUID="34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 05:28:56.258870 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:56.258795    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:28:56.830249 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:28:56.830212    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:28:56.987583 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:56.987540    5898 patch_prober.go:29] interesting pod/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 05:28:56.988126 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:56.987706    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw" podUID="34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 05:28:57.991647 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:57.991613    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw"
Oct 30 05:28:57.992221 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:57.992014    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw"
Oct 30 05:28:57.992847 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:57.992823    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw"
Oct 30 05:28:58.010809 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:58.010788    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:28:58.114699 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:58.114660    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:28:58.252222 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:28:58.252080    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:29:00.068052 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:00.068022    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:29:01.763061 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:01.763033    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:29:03.046692 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:03.046648    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:29:03.639500 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:03.639469    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:29:06.398234 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:06.398194    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:29:08.381678 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:08.381641    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:29:10.234527 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:10.234496    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:29:12.185615 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:12.185582    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:29:13.418282 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:13.418236    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:29:13.978090 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:13.978050    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:29:15.704322 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:15.704274    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:29:17.476038 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:17.475996    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:29:18.864219 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:18.864188    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54"]
Oct 30 05:29:18.870175 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:18.870152    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54"
Oct 30 05:29:18.911952 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:18.897062    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-9dc17a-openshift-default-dockercfg-k442k\""
Oct 30 05:29:18.911952 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:18.906978    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7"]
Oct 30 05:29:18.912434 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:18.912412    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54"]
Oct 30 05:29:18.912544 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:18.912506    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7"
Oct 30 05:29:18.928535 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:18.928508    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7"]
Oct 30 05:29:18.997801 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:18.997766    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b39e7705-286b-4d9a-8d3a-41dbfa9e20f2-istiod-ca-cert\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7\" (UID: \"b39e7705-286b-4d9a-8d3a-41dbfa9e20f2\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7"
Oct 30 05:29:18.997977 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:18.997811    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs2rh\" (UniqueName: \"kubernetes.io/projected/033d7e0d-2790-46af-829c-78374de43cd1-kube-api-access-hs2rh\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54\" (UID: \"033d7e0d-2790-46af-829c-78374de43cd1\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54"
Oct 30 05:29:18.997977 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:18.997867    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39e7705-286b-4d9a-8d3a-41dbfa9e20f2-credential-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7\" (UID: \"b39e7705-286b-4d9a-8d3a-41dbfa9e20f2\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7"
Oct 30 05:29:18.998066 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:18.997996    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/033d7e0d-2790-46af-829c-78374de43cd1-credential-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54\" (UID: \"033d7e0d-2790-46af-829c-78374de43cd1\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54"
Oct 30 05:29:18.998066 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:18.998020    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/033d7e0d-2790-46af-829c-78374de43cd1-workload-certs\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54\" (UID: \"033d7e0d-2790-46af-829c-78374de43cd1\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54"
Oct 30 05:29:18.998066 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:18.998045    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/033d7e0d-2790-46af-829c-78374de43cd1-istio-data\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54\" (UID: \"033d7e0d-2790-46af-829c-78374de43cd1\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54"
Oct 30 05:29:18.998173 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:18.998107    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/033d7e0d-2790-46af-829c-78374de43cd1-workload-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54\" (UID: \"033d7e0d-2790-46af-829c-78374de43cd1\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54"
Oct 30 05:29:18.998173 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:18.998135    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/033d7e0d-2790-46af-829c-78374de43cd1-istio-token\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54\" (UID: \"033d7e0d-2790-46af-829c-78374de43cd1\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54"
Oct 30 05:29:18.998258 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:18.998238    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/033d7e0d-2790-46af-829c-78374de43cd1-istio-envoy\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54\" (UID: \"033d7e0d-2790-46af-829c-78374de43cd1\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54"
Oct 30 05:29:18.998310 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:18.998299    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b39e7705-286b-4d9a-8d3a-41dbfa9e20f2-istio-envoy\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7\" (UID: \"b39e7705-286b-4d9a-8d3a-41dbfa9e20f2\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7"
Oct 30 05:29:18.998350 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:18.998328    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/033d7e0d-2790-46af-829c-78374de43cd1-istiod-ca-cert\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54\" (UID: \"033d7e0d-2790-46af-829c-78374de43cd1\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54"
Oct 30 05:29:18.998386 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:18.998353    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/033d7e0d-2790-46af-829c-78374de43cd1-istio-podinfo\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54\" (UID: \"033d7e0d-2790-46af-829c-78374de43cd1\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54"
Oct 30 05:29:18.998434 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:18.998421    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b39e7705-286b-4d9a-8d3a-41dbfa9e20f2-workload-certs\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7\" (UID: \"b39e7705-286b-4d9a-8d3a-41dbfa9e20f2\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7"
Oct 30 05:29:18.998471 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:18.998455    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b39e7705-286b-4d9a-8d3a-41dbfa9e20f2-istio-podinfo\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7\" (UID: \"b39e7705-286b-4d9a-8d3a-41dbfa9e20f2\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7"
Oct 30 05:29:18.998505 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:18.998491    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39e7705-286b-4d9a-8d3a-41dbfa9e20f2-workload-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7\" (UID: \"b39e7705-286b-4d9a-8d3a-41dbfa9e20f2\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7"
Oct 30 05:29:18.998540 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:18.998530    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4pzq\" (UniqueName: \"kubernetes.io/projected/b39e7705-286b-4d9a-8d3a-41dbfa9e20f2-kube-api-access-k4pzq\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7\" (UID: \"b39e7705-286b-4d9a-8d3a-41dbfa9e20f2\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7"
Oct 30 05:29:18.998656 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:18.998643    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b39e7705-286b-4d9a-8d3a-41dbfa9e20f2-istio-token\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7\" (UID: \"b39e7705-286b-4d9a-8d3a-41dbfa9e20f2\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7"
Oct 30 05:29:18.998698 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:18.998683    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b39e7705-286b-4d9a-8d3a-41dbfa9e20f2-istio-data\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7\" (UID: \"b39e7705-286b-4d9a-8d3a-41dbfa9e20f2\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7"
Oct 30 05:29:19.100128 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:19.100095    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/033d7e0d-2790-46af-829c-78374de43cd1-istio-podinfo\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54\" (UID: \"033d7e0d-2790-46af-829c-78374de43cd1\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54"
Oct 30 05:29:19.100307 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:19.100141    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b39e7705-286b-4d9a-8d3a-41dbfa9e20f2-workload-certs\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7\" (UID: \"b39e7705-286b-4d9a-8d3a-41dbfa9e20f2\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7"
Oct 30 05:29:19.100307 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:19.100280    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b39e7705-286b-4d9a-8d3a-41dbfa9e20f2-istio-podinfo\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7\" (UID: \"b39e7705-286b-4d9a-8d3a-41dbfa9e20f2\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7"
Oct 30 05:29:19.100402 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:19.100344    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39e7705-286b-4d9a-8d3a-41dbfa9e20f2-workload-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7\" (UID: \"b39e7705-286b-4d9a-8d3a-41dbfa9e20f2\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7"
Oct 30 05:29:19.100402 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:19.100378    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4pzq\" (UniqueName: \"kubernetes.io/projected/b39e7705-286b-4d9a-8d3a-41dbfa9e20f2-kube-api-access-k4pzq\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7\" (UID: \"b39e7705-286b-4d9a-8d3a-41dbfa9e20f2\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7"
Oct 30 05:29:19.100499 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:19.100474    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b39e7705-286b-4d9a-8d3a-41dbfa9e20f2-istio-token\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7\" (UID: \"b39e7705-286b-4d9a-8d3a-41dbfa9e20f2\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7"
Oct 30 05:29:19.100553 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:19.100522    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b39e7705-286b-4d9a-8d3a-41dbfa9e20f2-istio-data\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7\" (UID: \"b39e7705-286b-4d9a-8d3a-41dbfa9e20f2\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7"
Oct 30 05:29:19.100608 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:19.100555    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b39e7705-286b-4d9a-8d3a-41dbfa9e20f2-workload-certs\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7\" (UID: \"b39e7705-286b-4d9a-8d3a-41dbfa9e20f2\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7"
Oct 30 05:29:19.100608 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:19.100560    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b39e7705-286b-4d9a-8d3a-41dbfa9e20f2-istiod-ca-cert\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7\" (UID: \"b39e7705-286b-4d9a-8d3a-41dbfa9e20f2\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7"
Oct 30 05:29:19.100716 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:19.100609    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs2rh\" (UniqueName: \"kubernetes.io/projected/033d7e0d-2790-46af-829c-78374de43cd1-kube-api-access-hs2rh\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54\" (UID: \"033d7e0d-2790-46af-829c-78374de43cd1\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54"
Oct 30 05:29:19.100716 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:19.100640    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39e7705-286b-4d9a-8d3a-41dbfa9e20f2-workload-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7\" (UID: \"b39e7705-286b-4d9a-8d3a-41dbfa9e20f2\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7"
Oct 30 05:29:19.100716 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:19.100646    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39e7705-286b-4d9a-8d3a-41dbfa9e20f2-credential-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7\" (UID: \"b39e7705-286b-4d9a-8d3a-41dbfa9e20f2\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7"
Oct 30 05:29:19.100892 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:19.100827    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/033d7e0d-2790-46af-829c-78374de43cd1-credential-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54\" (UID: \"033d7e0d-2790-46af-829c-78374de43cd1\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54"
Oct 30 05:29:19.100986 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:19.100891    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/033d7e0d-2790-46af-829c-78374de43cd1-workload-certs\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54\" (UID: \"033d7e0d-2790-46af-829c-78374de43cd1\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54"
Oct 30 05:29:19.100986 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:19.100958    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/033d7e0d-2790-46af-829c-78374de43cd1-istio-data\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54\" (UID: \"033d7e0d-2790-46af-829c-78374de43cd1\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54"
Oct 30 05:29:19.101097 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:19.100989    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/033d7e0d-2790-46af-829c-78374de43cd1-workload-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54\" (UID: \"033d7e0d-2790-46af-829c-78374de43cd1\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54"
Oct 30 05:29:19.101097 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:19.101015    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/033d7e0d-2790-46af-829c-78374de43cd1-istio-token\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54\" (UID: \"033d7e0d-2790-46af-829c-78374de43cd1\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54"
Oct 30 05:29:19.101097 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:19.101079    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/033d7e0d-2790-46af-829c-78374de43cd1-istio-envoy\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54\" (UID: \"033d7e0d-2790-46af-829c-78374de43cd1\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54"
Oct 30 05:29:19.101275 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:19.101102    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b39e7705-286b-4d9a-8d3a-41dbfa9e20f2-istio-data\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7\" (UID: \"b39e7705-286b-4d9a-8d3a-41dbfa9e20f2\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7"
Oct 30 05:29:19.101275 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:19.101179    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b39e7705-286b-4d9a-8d3a-41dbfa9e20f2-istio-envoy\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7\" (UID: \"b39e7705-286b-4d9a-8d3a-41dbfa9e20f2\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7"
Oct 30 05:29:19.101275 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:19.101229    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/033d7e0d-2790-46af-829c-78374de43cd1-istiod-ca-cert\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54\" (UID: \"033d7e0d-2790-46af-829c-78374de43cd1\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54"
Oct 30 05:29:19.101275 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:19.101261    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39e7705-286b-4d9a-8d3a-41dbfa9e20f2-credential-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7\" (UID: \"b39e7705-286b-4d9a-8d3a-41dbfa9e20f2\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7"
Oct 30 05:29:19.101504 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:19.101260    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b39e7705-286b-4d9a-8d3a-41dbfa9e20f2-istiod-ca-cert\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7\" (UID: \"b39e7705-286b-4d9a-8d3a-41dbfa9e20f2\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7"
Oct 30 05:29:19.101504 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:19.101404    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/033d7e0d-2790-46af-829c-78374de43cd1-workload-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54\" (UID: \"033d7e0d-2790-46af-829c-78374de43cd1\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54"
Oct 30 05:29:19.101604 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:19.101493    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/033d7e0d-2790-46af-829c-78374de43cd1-credential-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54\" (UID: \"033d7e0d-2790-46af-829c-78374de43cd1\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54"
Oct 30 05:29:19.101950 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:19.101925    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/033d7e0d-2790-46af-829c-78374de43cd1-workload-certs\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54\" (UID: \"033d7e0d-2790-46af-829c-78374de43cd1\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54"
Oct 30 05:29:19.102417 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:19.102388    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/033d7e0d-2790-46af-829c-78374de43cd1-istio-data\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54\" (UID: \"033d7e0d-2790-46af-829c-78374de43cd1\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54"
Oct 30 05:29:19.102767 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:19.102708    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/033d7e0d-2790-46af-829c-78374de43cd1-istiod-ca-cert\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54\" (UID: \"033d7e0d-2790-46af-829c-78374de43cd1\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54"
Oct 30 05:29:19.103177 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:19.103159    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/033d7e0d-2790-46af-829c-78374de43cd1-istio-podinfo\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54\" (UID: \"033d7e0d-2790-46af-829c-78374de43cd1\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54"
Oct 30 05:29:19.103282 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:19.103263    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b39e7705-286b-4d9a-8d3a-41dbfa9e20f2-istio-podinfo\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7\" (UID: \"b39e7705-286b-4d9a-8d3a-41dbfa9e20f2\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7"
Oct 30 05:29:19.104179 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:19.104155    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b39e7705-286b-4d9a-8d3a-41dbfa9e20f2-istio-envoy\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7\" (UID: \"b39e7705-286b-4d9a-8d3a-41dbfa9e20f2\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7"
Oct 30 05:29:19.104414 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:19.104393    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/033d7e0d-2790-46af-829c-78374de43cd1-istio-envoy\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54\" (UID: \"033d7e0d-2790-46af-829c-78374de43cd1\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54"
Oct 30 05:29:19.121543 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:19.121478    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs2rh\" (UniqueName: \"kubernetes.io/projected/033d7e0d-2790-46af-829c-78374de43cd1-kube-api-access-hs2rh\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54\" (UID: \"033d7e0d-2790-46af-829c-78374de43cd1\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54"
Oct 30 05:29:19.121659 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:19.121637    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4pzq\" (UniqueName: \"kubernetes.io/projected/b39e7705-286b-4d9a-8d3a-41dbfa9e20f2-kube-api-access-k4pzq\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7\" (UID: \"b39e7705-286b-4d9a-8d3a-41dbfa9e20f2\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7"
Oct 30 05:29:19.121734 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:19.121719    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/033d7e0d-2790-46af-829c-78374de43cd1-istio-token\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54\" (UID: \"033d7e0d-2790-46af-829c-78374de43cd1\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54"
Oct 30 05:29:19.122140 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:19.122123    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b39e7705-286b-4d9a-8d3a-41dbfa9e20f2-istio-token\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7\" (UID: \"b39e7705-286b-4d9a-8d3a-41dbfa9e20f2\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7"
Oct 30 05:29:19.185751 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:19.185717    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54"
Oct 30 05:29:19.245445 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:19.235335    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7"
Oct 30 05:29:19.515024 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:19.514645    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:29:19.801491 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:19.801464    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7"]
Oct 30 05:29:19.802559 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:29:19.802532    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb39e7705_286b_4d9a_8d3a_41dbfa9e20f2.slice/crio-c00fa7e5a1104f94bf2259f698893beaef8541da477c84aba66d1f72ccf6b16f WatchSource:0}: Error finding container c00fa7e5a1104f94bf2259f698893beaef8541da477c84aba66d1f72ccf6b16f: Status 404 returned error can't find the container with id c00fa7e5a1104f94bf2259f698893beaef8541da477c84aba66d1f72ccf6b16f
Oct 30 05:29:19.803457 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:29:19.803433    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod033d7e0d_2790_46af_829c_78374de43cd1.slice/crio-a816dd887f1492ff6e042edce2b47163af3595d070a7ddfb182ac8b75d2f6010 WatchSource:0}: Error finding container a816dd887f1492ff6e042edce2b47163af3595d070a7ddfb182ac8b75d2f6010: Status 404 returned error can't find the container with id a816dd887f1492ff6e042edce2b47163af3595d070a7ddfb182ac8b75d2f6010
Oct 30 05:29:19.805358 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:19.805327    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:29:19.805418 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:19.805402    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:29:19.805453 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:19.805438    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:29:19.806309 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:19.806280    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:29:19.806402 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:19.806340    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:29:19.806402 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:19.806370    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:29:19.811571 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:19.811543    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54"]
Oct 30 05:29:20.680009 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:20.678294    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7" event={"ID":"b39e7705-286b-4d9a-8d3a-41dbfa9e20f2","Type":"ContainerStarted","Data":"62cce9c2945e944d0b4851cc15b0904e970c4a8ee45585b94354f5f6b026113c"}
Oct 30 05:29:20.680009 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:20.678366    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7" event={"ID":"b39e7705-286b-4d9a-8d3a-41dbfa9e20f2","Type":"ContainerStarted","Data":"c00fa7e5a1104f94bf2259f698893beaef8541da477c84aba66d1f72ccf6b16f"}
Oct 30 05:29:20.683029 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:20.682991    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54" event={"ID":"033d7e0d-2790-46af-829c-78374de43cd1","Type":"ContainerStarted","Data":"2143370d428514c87be5c439b81b6fcd172b9051c43146d903a849096688c2f7"}
Oct 30 05:29:20.683277 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:20.683252    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54" event={"ID":"033d7e0d-2790-46af-829c-78374de43cd1","Type":"ContainerStarted","Data":"a816dd887f1492ff6e042edce2b47163af3595d070a7ddfb182ac8b75d2f6010"}
Oct 30 05:29:20.743627 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:20.743570    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7" podStartSLOduration=2.743548212 podStartE2EDuration="2.743548212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:29:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:29:20.743231605 +0000 UTC m=+2689.967051529" watchObservedRunningTime="2026-10-30 05:29:20.743548212 +0000 UTC m=+2689.967368112"
Oct 30 05:29:20.807926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:20.807554    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54" podStartSLOduration=2.807532956 podStartE2EDuration="2.807532956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:29:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:29:20.7853167 +0000 UTC m=+2690.009136630" watchObservedRunningTime="2026-10-30 05:29:20.807532956 +0000 UTC m=+2690.031352858"
Oct 30 05:29:21.206028 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:21.205978    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54"
Oct 30 05:29:21.210026 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:21.209993    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Oct 30 05:29:21.210272 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:21.210247    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54" podUID="033d7e0d-2790-46af-829c-78374de43cd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Oct 30 05:29:21.239555 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:21.239524    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7"
Oct 30 05:29:21.241152 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:21.241116    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Oct 30 05:29:21.241288 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:21.241179    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7" podUID="b39e7705-286b-4d9a-8d3a-41dbfa9e20f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Oct 30 05:29:21.430670 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:21.430630    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:29:22.228950 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:22.209448    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Oct 30 05:29:22.228950 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:22.209529    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54" podUID="033d7e0d-2790-46af-829c-78374de43cd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Oct 30 05:29:22.239397 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:22.239366    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Oct 30 05:29:22.239663 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:22.239640    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7" podUID="b39e7705-286b-4d9a-8d3a-41dbfa9e20f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Oct 30 05:29:22.372124 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:22.372090    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:29:22.472753 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:22.472716    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:29:22.600938 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:22.583234    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:29:23.192929 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:23.190371    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Oct 30 05:29:23.192929 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:23.190441    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54" podUID="033d7e0d-2790-46af-829c-78374de43cd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Oct 30 05:29:23.239926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:23.239343    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Oct 30 05:29:23.239926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:23.239415    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7" podUID="b39e7705-286b-4d9a-8d3a-41dbfa9e20f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Oct 30 05:29:23.995927 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:23.989511    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:29:24.207925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:24.195998    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Oct 30 05:29:24.207925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:24.196072    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54" podUID="033d7e0d-2790-46af-829c-78374de43cd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Oct 30 05:29:24.239509 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:24.239477    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Oct 30 05:29:24.239690 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:24.239532    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7" podUID="b39e7705-286b-4d9a-8d3a-41dbfa9e20f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Oct 30 05:29:25.189927 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:25.188261    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Oct 30 05:29:25.189927 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:25.188321    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54" podUID="033d7e0d-2790-46af-829c-78374de43cd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Oct 30 05:29:25.239423 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:25.239386    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Oct 30 05:29:25.239620 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:25.239454    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7" podUID="b39e7705-286b-4d9a-8d3a-41dbfa9e20f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Oct 30 05:29:25.829848 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:25.829810    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:29:26.200838 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:26.200750    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Oct 30 05:29:26.201473 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:26.201439    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54" podUID="033d7e0d-2790-46af-829c-78374de43cd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Oct 30 05:29:26.239365 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:26.239331    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Oct 30 05:29:26.239521 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:26.239397    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7" podUID="b39e7705-286b-4d9a-8d3a-41dbfa9e20f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Oct 30 05:29:26.671705 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:26.667221    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:29:26.833642 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:29:26.833595    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:29:27.296850 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:27.296817    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54"
Oct 30 05:29:27.297919 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:27.297885    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54"
Oct 30 05:29:27.301022 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:27.301004    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54"
Oct 30 05:29:27.317385 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:27.317354    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7"
Oct 30 05:29:27.317758 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:27.317737    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7"
Oct 30 05:29:27.318450 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:27.318431    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7"
Oct 30 05:29:27.864355 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:27.864310    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:29:29.739550 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:29.739520    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:29:31.460756 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:31.460718    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:29:31.825623 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:31.825592    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h"]
Oct 30 05:29:31.831082 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:31.831060    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h"
Oct 30 05:29:31.846411 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:31.846386    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h"]
Oct 30 05:29:31.851950 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:31.851924    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-0dba31-openshift-default-dockercfg-9kc5b\""
Oct 30 05:29:31.939131 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:31.939097    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/748d8cba-95b7-49de-8f41-d47a8876df1f-istio-data\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h\" (UID: \"748d8cba-95b7-49de-8f41-d47a8876df1f\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h"
Oct 30 05:29:31.939317 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:31.939159    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/748d8cba-95b7-49de-8f41-d47a8876df1f-workload-certs\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h\" (UID: \"748d8cba-95b7-49de-8f41-d47a8876df1f\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h"
Oct 30 05:29:31.939317 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:31.939266    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/748d8cba-95b7-49de-8f41-d47a8876df1f-istiod-ca-cert\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h\" (UID: \"748d8cba-95b7-49de-8f41-d47a8876df1f\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h"
Oct 30 05:29:31.939459 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:31.939319    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/748d8cba-95b7-49de-8f41-d47a8876df1f-istio-token\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h\" (UID: \"748d8cba-95b7-49de-8f41-d47a8876df1f\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h"
Oct 30 05:29:31.939589 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:31.939484    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/748d8cba-95b7-49de-8f41-d47a8876df1f-credential-socket\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h\" (UID: \"748d8cba-95b7-49de-8f41-d47a8876df1f\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h"
Oct 30 05:29:31.939691 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:31.939674    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/748d8cba-95b7-49de-8f41-d47a8876df1f-istio-podinfo\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h\" (UID: \"748d8cba-95b7-49de-8f41-d47a8876df1f\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h"
Oct 30 05:29:31.939753 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:31.939715    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgpn9\" (UniqueName: \"kubernetes.io/projected/748d8cba-95b7-49de-8f41-d47a8876df1f-kube-api-access-qgpn9\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h\" (UID: \"748d8cba-95b7-49de-8f41-d47a8876df1f\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h"
Oct 30 05:29:31.939816 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:31.939800    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/748d8cba-95b7-49de-8f41-d47a8876df1f-workload-socket\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h\" (UID: \"748d8cba-95b7-49de-8f41-d47a8876df1f\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h"
Oct 30 05:29:31.939949 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:31.939895    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/748d8cba-95b7-49de-8f41-d47a8876df1f-istio-envoy\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h\" (UID: \"748d8cba-95b7-49de-8f41-d47a8876df1f\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h"
Oct 30 05:29:32.041437 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:32.041402    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/748d8cba-95b7-49de-8f41-d47a8876df1f-istio-envoy\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h\" (UID: \"748d8cba-95b7-49de-8f41-d47a8876df1f\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h"
Oct 30 05:29:32.041623 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:32.041467    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/748d8cba-95b7-49de-8f41-d47a8876df1f-istio-data\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h\" (UID: \"748d8cba-95b7-49de-8f41-d47a8876df1f\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h"
Oct 30 05:29:32.041623 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:32.041523    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/748d8cba-95b7-49de-8f41-d47a8876df1f-workload-certs\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h\" (UID: \"748d8cba-95b7-49de-8f41-d47a8876df1f\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h"
Oct 30 05:29:32.041623 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:32.041564    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/748d8cba-95b7-49de-8f41-d47a8876df1f-istiod-ca-cert\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h\" (UID: \"748d8cba-95b7-49de-8f41-d47a8876df1f\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h"
Oct 30 05:29:32.041623 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:32.041605    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/748d8cba-95b7-49de-8f41-d47a8876df1f-istio-token\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h\" (UID: \"748d8cba-95b7-49de-8f41-d47a8876df1f\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h"
Oct 30 05:29:32.041855 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:32.041701    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/748d8cba-95b7-49de-8f41-d47a8876df1f-credential-socket\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h\" (UID: \"748d8cba-95b7-49de-8f41-d47a8876df1f\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h"
Oct 30 05:29:32.041952 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:32.041861    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/748d8cba-95b7-49de-8f41-d47a8876df1f-istio-podinfo\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h\" (UID: \"748d8cba-95b7-49de-8f41-d47a8876df1f\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h"
Oct 30 05:29:32.042083 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:32.042060    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgpn9\" (UniqueName: \"kubernetes.io/projected/748d8cba-95b7-49de-8f41-d47a8876df1f-kube-api-access-qgpn9\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h\" (UID: \"748d8cba-95b7-49de-8f41-d47a8876df1f\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h"
Oct 30 05:29:32.042165 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:32.042130    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/748d8cba-95b7-49de-8f41-d47a8876df1f-workload-socket\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h\" (UID: \"748d8cba-95b7-49de-8f41-d47a8876df1f\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h"
Oct 30 05:29:32.043152 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:32.042184    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/748d8cba-95b7-49de-8f41-d47a8876df1f-istio-data\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h\" (UID: \"748d8cba-95b7-49de-8f41-d47a8876df1f\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h"
Oct 30 05:29:32.043152 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:32.042267    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/748d8cba-95b7-49de-8f41-d47a8876df1f-credential-socket\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h\" (UID: \"748d8cba-95b7-49de-8f41-d47a8876df1f\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h"
Oct 30 05:29:32.043152 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:32.042520    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/748d8cba-95b7-49de-8f41-d47a8876df1f-workload-socket\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h\" (UID: \"748d8cba-95b7-49de-8f41-d47a8876df1f\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h"
Oct 30 05:29:32.043391 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:32.043230    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/748d8cba-95b7-49de-8f41-d47a8876df1f-istiod-ca-cert\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h\" (UID: \"748d8cba-95b7-49de-8f41-d47a8876df1f\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h"
Oct 30 05:29:32.043499 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:32.043470    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/748d8cba-95b7-49de-8f41-d47a8876df1f-workload-certs\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h\" (UID: \"748d8cba-95b7-49de-8f41-d47a8876df1f\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h"
Oct 30 05:29:32.044676 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:32.044654    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/748d8cba-95b7-49de-8f41-d47a8876df1f-istio-envoy\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h\" (UID: \"748d8cba-95b7-49de-8f41-d47a8876df1f\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h"
Oct 30 05:29:32.044935 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:32.044918    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/748d8cba-95b7-49de-8f41-d47a8876df1f-istio-podinfo\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h\" (UID: \"748d8cba-95b7-49de-8f41-d47a8876df1f\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h"
Oct 30 05:29:32.061768 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:32.061738    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgpn9\" (UniqueName: \"kubernetes.io/projected/748d8cba-95b7-49de-8f41-d47a8876df1f-kube-api-access-qgpn9\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h\" (UID: \"748d8cba-95b7-49de-8f41-d47a8876df1f\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h"
Oct 30 05:29:32.061768 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:32.061757    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/748d8cba-95b7-49de-8f41-d47a8876df1f-istio-token\") pod \"e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h\" (UID: \"748d8cba-95b7-49de-8f41-d47a8876df1f\") " pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h"
Oct 30 05:29:32.148236 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:32.148160    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h"
Oct 30 05:29:32.472203 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:32.472178    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h"]
Oct 30 05:29:32.474607 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:29:32.474580    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod748d8cba_95b7_49de_8f41_d47a8876df1f.slice/crio-c6607470134f3f971f826fb240d808336f1f922f523185bd7f1d66a6ce7068a1 WatchSource:0}: Error finding container c6607470134f3f971f826fb240d808336f1f922f523185bd7f1d66a6ce7068a1: Status 404 returned error can't find the container with id c6607470134f3f971f826fb240d808336f1f922f523185bd7f1d66a6ce7068a1
Oct 30 05:29:32.477779 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:32.477747    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:29:32.477862 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:32.477820    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:29:32.477862 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:32.477853    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:29:32.931481 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:32.931447    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h" event={"ID":"748d8cba-95b7-49de-8f41-d47a8876df1f","Type":"ContainerStarted","Data":"64c16e97dd25e311310e6af85491b50139b687b55f66e4372af15792acdc56b5"}
Oct 30 05:29:32.931691 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:32.931500    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h" event={"ID":"748d8cba-95b7-49de-8f41-d47a8876df1f","Type":"ContainerStarted","Data":"c6607470134f3f971f826fb240d808336f1f922f523185bd7f1d66a6ce7068a1"}
Oct 30 05:29:33.148565 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:33.148531    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h"
Oct 30 05:29:33.150241 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:33.150210    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Oct 30 05:29:33.150373 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:33.150261    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h" podUID="748d8cba-95b7-49de-8f41-d47a8876df1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Oct 30 05:29:33.395924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:33.374769    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:29:34.148772 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:34.148735    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Oct 30 05:29:34.149323 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:34.148799    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h" podUID="748d8cba-95b7-49de-8f41-d47a8876df1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Oct 30 05:29:35.121503 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:35.121474    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:29:35.148931 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:35.148885    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Oct 30 05:29:35.149374 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:35.148952    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h" podUID="748d8cba-95b7-49de-8f41-d47a8876df1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Oct 30 05:29:35.506766 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:35.506647    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:29:36.149451 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:36.149416    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Oct 30 05:29:36.149831 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:36.149468    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h" podUID="748d8cba-95b7-49de-8f41-d47a8876df1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Oct 30 05:29:36.951517 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:36.951474    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:29:37.148748 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:37.148716    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Oct 30 05:29:37.148954 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:37.148770    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h" podUID="748d8cba-95b7-49de-8f41-d47a8876df1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Oct 30 05:29:38.148639 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:38.148603    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Oct 30 05:29:38.149056 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:38.148680    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h" podUID="748d8cba-95b7-49de-8f41-d47a8876df1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Oct 30 05:29:38.963816 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:38.963785    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:29:39.244105 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:39.244022    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h"
Oct 30 05:29:39.245065 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:39.245027    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h"
Oct 30 05:29:39.246182 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:39.246159    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h"
Oct 30 05:29:39.313856 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:39.313807    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h" podStartSLOduration=8.313794521 podStartE2EDuration="8.313794521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:29:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:29:33.000498269 +0000 UTC m=+2702.224318200" watchObservedRunningTime="2026-10-30 05:29:39.313794521 +0000 UTC m=+2708.537614422"
Oct 30 05:29:40.789207 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:40.789170    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:29:42.546466 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:42.546384    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:29:42.867183 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:42.867152    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:29:44.103177 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:44.103144    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:29:45.120096 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:45.120062    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:29:46.528848 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:46.528813    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:29:47.469944 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:47.469882    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:29:49.132143 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:49.132105    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:29:51.169213 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:51.169152    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:29:52.932615 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:52.932561    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:29:54.040997 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:54.040958    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-93bad9/rate-test-model-93bad9-kserve-6878b4bf49-hcjsm"]
Oct 30 05:29:54.100149 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:54.100101    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:29:54.735330 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:54.735296    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:29:54.962210 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:54.962178    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:29:55.274692 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:55.274655    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:29:55.464923 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:55.463185    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:29:55.601100 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:55.601063    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:29:56.819666 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:29:56.819632    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:29:57.046536 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:57.046499    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:29:59.080844 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:29:59.080811    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:00.098215 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:00.094950    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:00.268886 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:00.268848    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2f6fed-29888970-rvzpv"]
Oct 30 05:30:00.304011 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:00.297807    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:00.320079 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:00.309111    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2f6fed-29888970-rvzpv"
Oct 30 05:30:00.340049 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:00.309018    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888970-k4phq"]
Oct 30 05:30:00.366064 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:00.344076    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\""
Oct 30 05:30:00.450258 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:00.450213    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888970-qhwpt"]
Oct 30 05:30:00.450938 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:00.450812    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888970-k4phq"
Oct 30 05:30:00.521923 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:00.479212    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbcc00ce-d37e-49fc-8755-7100bdfdd954-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-2f6fed-29888970-rvzpv\" (UID: \"fbcc00ce-d37e-49fc-8755-7100bdfdd954\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2f6fed-29888970-rvzpv"
Oct 30 05:30:00.521923 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:00.483604    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlp4d\" (UniqueName: \"kubernetes.io/projected/fbcc00ce-d37e-49fc-8755-7100bdfdd954-kube-api-access-xlp4d\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-2f6fed-29888970-rvzpv\" (UID: \"fbcc00ce-d37e-49fc-8755-7100bdfdd954\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2f6fed-29888970-rvzpv"
Oct 30 05:30:00.521923 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:00.514062    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888970-mf2ct"]
Oct 30 05:30:00.521923 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:00.517962    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888970-qhwpt"
Oct 30 05:30:00.529222 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:00.522541    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2f6fed-29888970-rvzpv"]
Oct 30 05:30:00.529222 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:00.522628    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888970-mf2ct"
Oct 30 05:30:00.529222 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:00.525022    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888970-hsjd5"]
Oct 30 05:30:00.534161 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:00.534135    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888970-hsjd5"
Oct 30 05:30:00.545751 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:00.538489    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888970-22ksr"]
Oct 30 05:30:00.577833 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:00.577802    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888970-k9zdq"]
Oct 30 05:30:00.598510 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:00.598483    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888970-22ksr"
Oct 30 05:30:00.610919 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:00.610277    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d8a177e-9a55-45dc-8b18-548547e72b6b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-83e83e-29888970-k4phq\" (UID: \"2d8a177e-9a55-45dc-8b18-548547e72b6b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888970-k4phq"
Oct 30 05:30:00.610919 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:00.610351    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f05b65d9-9992-445b-af08-28759a4ef2e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888970-mf2ct\" (UID: \"f05b65d9-9992-445b-af08-28759a4ef2e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888970-mf2ct"
Oct 30 05:30:00.610919 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:00.610445    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08f30f4b-4298-494f-9fe3-de259a1991a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888970-qhwpt\" (UID: \"08f30f4b-4298-494f-9fe3-de259a1991a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888970-qhwpt"
Oct 30 05:30:00.610919 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:00.610484    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbcc00ce-d37e-49fc-8755-7100bdfdd954-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-2f6fed-29888970-rvzpv\" (UID: \"fbcc00ce-d37e-49fc-8755-7100bdfdd954\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2f6fed-29888970-rvzpv"
Oct 30 05:30:00.610919 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:00.610600    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlp4d\" (UniqueName: \"kubernetes.io/projected/fbcc00ce-d37e-49fc-8755-7100bdfdd954-kube-api-access-xlp4d\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-2f6fed-29888970-rvzpv\" (UID: \"fbcc00ce-d37e-49fc-8755-7100bdfdd954\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2f6fed-29888970-rvzpv"
Oct 30 05:30:00.610919 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:00.610869    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjvnw\" (UniqueName: \"kubernetes.io/projected/f05b65d9-9992-445b-af08-28759a4ef2e2-kube-api-access-mjvnw\") pod \"maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888970-mf2ct\" (UID: \"f05b65d9-9992-445b-af08-28759a4ef2e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888970-mf2ct"
Oct 30 05:30:00.611324 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:00.610952    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jzd9\" (UniqueName: \"kubernetes.io/projected/08f30f4b-4298-494f-9fe3-de259a1991a0-kube-api-access-6jzd9\") pod \"maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888970-qhwpt\" (UID: \"08f30f4b-4298-494f-9fe3-de259a1991a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888970-qhwpt"
Oct 30 05:30:00.611324 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:00.611017    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxqtq\" (UniqueName: \"kubernetes.io/projected/2d8a177e-9a55-45dc-8b18-548547e72b6b-kube-api-access-qxqtq\") pod \"maas-api-key-cleanup-e2e-worker-w6-83e83e-29888970-k4phq\" (UID: \"2d8a177e-9a55-45dc-8b18-548547e72b6b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888970-k4phq"
Oct 30 05:30:00.629834 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:00.629291    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbcc00ce-d37e-49fc-8755-7100bdfdd954-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-2f6fed-29888970-rvzpv\" (UID: \"fbcc00ce-d37e-49fc-8755-7100bdfdd954\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2f6fed-29888970-rvzpv"
Oct 30 05:30:00.639872 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:00.631831    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888970-5ssdn"]
Oct 30 05:30:00.639872 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:00.632567    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888970-k9zdq"
Oct 30 05:30:00.672923 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:00.672182    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888970-qhwpt"]
Oct 30 05:30:00.672923 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:00.672250    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888970-k4phq"]
Oct 30 05:30:00.672923 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:00.672384    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888970-5ssdn"
Oct 30 05:30:00.719410 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:00.719075    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jzd9\" (UniqueName: \"kubernetes.io/projected/08f30f4b-4298-494f-9fe3-de259a1991a0-kube-api-access-6jzd9\") pod \"maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888970-qhwpt\" (UID: \"08f30f4b-4298-494f-9fe3-de259a1991a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888970-qhwpt"
Oct 30 05:30:00.719410 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:00.719148    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxqtq\" (UniqueName: \"kubernetes.io/projected/2d8a177e-9a55-45dc-8b18-548547e72b6b-kube-api-access-qxqtq\") pod \"maas-api-key-cleanup-e2e-worker-w6-83e83e-29888970-k4phq\" (UID: \"2d8a177e-9a55-45dc-8b18-548547e72b6b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888970-k4phq"
Oct 30 05:30:00.719410 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:00.719234    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d8a177e-9a55-45dc-8b18-548547e72b6b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-83e83e-29888970-k4phq\" (UID: \"2d8a177e-9a55-45dc-8b18-548547e72b6b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888970-k4phq"
Oct 30 05:30:00.719410 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:00.719281    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f05b65d9-9992-445b-af08-28759a4ef2e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888970-mf2ct\" (UID: \"f05b65d9-9992-445b-af08-28759a4ef2e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888970-mf2ct"
Oct 30 05:30:00.719410 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:00.719344    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8988\" (UniqueName: \"kubernetes.io/projected/b49de578-c4c0-4ae7-a090-4b9d8695b62c-kube-api-access-b8988\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888970-hsjd5\" (UID: \"b49de578-c4c0-4ae7-a090-4b9d8695b62c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888970-hsjd5"
Oct 30 05:30:00.719410 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:00.719399    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dd199d3-31ed-41c5-8570-f6a1bc1cbdc9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888970-k9zdq\" (UID: \"7dd199d3-31ed-41c5-8570-f6a1bc1cbdc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888970-k9zdq"
Oct 30 05:30:00.719796 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:00.719434    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08f30f4b-4298-494f-9fe3-de259a1991a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888970-qhwpt\" (UID: \"08f30f4b-4298-494f-9fe3-de259a1991a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888970-qhwpt"
Oct 30 05:30:00.719796 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:00.719522    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b49de578-c4c0-4ae7-a090-4b9d8695b62c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888970-hsjd5\" (UID: \"b49de578-c4c0-4ae7-a090-4b9d8695b62c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888970-hsjd5"
Oct 30 05:30:00.719796 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:00.719562    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxqhk\" (UniqueName: \"kubernetes.io/projected/c2b36ef9-dd65-43f2-88fe-f360f18ba0fc-kube-api-access-wxqhk\") pod \"maas-api-key-cleanup-e2e-worker-w3-889ca9-29888970-22ksr\" (UID: \"c2b36ef9-dd65-43f2-88fe-f360f18ba0fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888970-22ksr"
Oct 30 05:30:00.719796 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:00.719764    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2b36ef9-dd65-43f2-88fe-f360f18ba0fc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-889ca9-29888970-22ksr\" (UID: \"c2b36ef9-dd65-43f2-88fe-f360f18ba0fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888970-22ksr"
Oct 30 05:30:00.720062 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:00.719881    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g22rz\" (UniqueName: \"kubernetes.io/projected/7dd199d3-31ed-41c5-8570-f6a1bc1cbdc9-kube-api-access-g22rz\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888970-k9zdq\" (UID: \"7dd199d3-31ed-41c5-8570-f6a1bc1cbdc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888970-k9zdq"
Oct 30 05:30:00.720062 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:00.719936    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjvnw\" (UniqueName: \"kubernetes.io/projected/f05b65d9-9992-445b-af08-28759a4ef2e2-kube-api-access-mjvnw\") pod \"maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888970-mf2ct\" (UID: \"f05b65d9-9992-445b-af08-28759a4ef2e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888970-mf2ct"
Oct 30 05:30:00.753926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:00.744320    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d8a177e-9a55-45dc-8b18-548547e72b6b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-83e83e-29888970-k4phq\" (UID: \"2d8a177e-9a55-45dc-8b18-548547e72b6b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888970-k4phq"
Oct 30 05:30:00.753926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:00.745197    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f05b65d9-9992-445b-af08-28759a4ef2e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888970-mf2ct\" (UID: \"f05b65d9-9992-445b-af08-28759a4ef2e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888970-mf2ct"
Oct 30 05:30:00.842067 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:00.799237    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08f30f4b-4298-494f-9fe3-de259a1991a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888970-qhwpt\" (UID: \"08f30f4b-4298-494f-9fe3-de259a1991a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888970-qhwpt"
Oct 30 05:30:00.934803 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:00.933055    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxqtq\" (UniqueName: \"kubernetes.io/projected/2d8a177e-9a55-45dc-8b18-548547e72b6b-kube-api-access-qxqtq\") pod \"maas-api-key-cleanup-e2e-worker-w6-83e83e-29888970-k4phq\" (UID: \"2d8a177e-9a55-45dc-8b18-548547e72b6b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888970-k4phq"
Oct 30 05:30:01.016626 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:00.972266    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888970-k4phq"
Oct 30 05:30:01.038293 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:01.037803    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jzd9\" (UniqueName: \"kubernetes.io/projected/08f30f4b-4298-494f-9fe3-de259a1991a0-kube-api-access-6jzd9\") pod \"maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888970-qhwpt\" (UID: \"08f30f4b-4298-494f-9fe3-de259a1991a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888970-qhwpt"
Oct 30 05:30:01.038496 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:01.038332    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8988\" (UniqueName: \"kubernetes.io/projected/b49de578-c4c0-4ae7-a090-4b9d8695b62c-kube-api-access-b8988\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888970-hsjd5\" (UID: \"b49de578-c4c0-4ae7-a090-4b9d8695b62c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888970-hsjd5"
Oct 30 05:30:01.038496 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:01.038391    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dd199d3-31ed-41c5-8570-f6a1bc1cbdc9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888970-k9zdq\" (UID: \"7dd199d3-31ed-41c5-8570-f6a1bc1cbdc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888970-k9zdq"
Oct 30 05:30:01.038496 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:01.038482    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czz75\" (UniqueName: \"kubernetes.io/projected/3ea1e4a0-ae9b-4a99-bf48-8f5a96e33e3b-kube-api-access-czz75\") pod \"maas-api-key-cleanup-e2e-worker-w0-b21507-29888970-5ssdn\" (UID: \"3ea1e4a0-ae9b-4a99-bf48-8f5a96e33e3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888970-5ssdn"
Oct 30 05:30:01.051058 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:01.038522    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b49de578-c4c0-4ae7-a090-4b9d8695b62c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888970-hsjd5\" (UID: \"b49de578-c4c0-4ae7-a090-4b9d8695b62c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888970-hsjd5"
Oct 30 05:30:01.051058 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:01.038557    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxqhk\" (UniqueName: \"kubernetes.io/projected/c2b36ef9-dd65-43f2-88fe-f360f18ba0fc-kube-api-access-wxqhk\") pod \"maas-api-key-cleanup-e2e-worker-w3-889ca9-29888970-22ksr\" (UID: \"c2b36ef9-dd65-43f2-88fe-f360f18ba0fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888970-22ksr"
Oct 30 05:30:01.051058 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:01.038586    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ea1e4a0-ae9b-4a99-bf48-8f5a96e33e3b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b21507-29888970-5ssdn\" (UID: \"3ea1e4a0-ae9b-4a99-bf48-8f5a96e33e3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888970-5ssdn"
Oct 30 05:30:01.051058 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:01.038787    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2b36ef9-dd65-43f2-88fe-f360f18ba0fc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-889ca9-29888970-22ksr\" (UID: \"c2b36ef9-dd65-43f2-88fe-f360f18ba0fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888970-22ksr"
Oct 30 05:30:01.051058 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:01.038954    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g22rz\" (UniqueName: \"kubernetes.io/projected/7dd199d3-31ed-41c5-8570-f6a1bc1cbdc9-kube-api-access-g22rz\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888970-k9zdq\" (UID: \"7dd199d3-31ed-41c5-8570-f6a1bc1cbdc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888970-k9zdq"
Oct 30 05:30:01.051058 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:01.039461    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dd199d3-31ed-41c5-8570-f6a1bc1cbdc9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888970-k9zdq\" (UID: \"7dd199d3-31ed-41c5-8570-f6a1bc1cbdc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888970-k9zdq"
Oct 30 05:30:01.051058 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:01.040074    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlp4d\" (UniqueName: \"kubernetes.io/projected/fbcc00ce-d37e-49fc-8755-7100bdfdd954-kube-api-access-xlp4d\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-2f6fed-29888970-rvzpv\" (UID: \"fbcc00ce-d37e-49fc-8755-7100bdfdd954\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2f6fed-29888970-rvzpv"
Oct 30 05:30:01.051058 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:01.040187    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b49de578-c4c0-4ae7-a090-4b9d8695b62c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888970-hsjd5\" (UID: \"b49de578-c4c0-4ae7-a090-4b9d8695b62c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888970-hsjd5"
Oct 30 05:30:01.051058 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:01.041142    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2b36ef9-dd65-43f2-88fe-f360f18ba0fc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-889ca9-29888970-22ksr\" (UID: \"c2b36ef9-dd65-43f2-88fe-f360f18ba0fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888970-22ksr"
Oct 30 05:30:01.051058 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:01.046544    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjvnw\" (UniqueName: \"kubernetes.io/projected/f05b65d9-9992-445b-af08-28759a4ef2e2-kube-api-access-mjvnw\") pod \"maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888970-mf2ct\" (UID: \"f05b65d9-9992-445b-af08-28759a4ef2e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888970-mf2ct"
Oct 30 05:30:01.051058 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:01.048876    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2f6fed-29888970-rvzpv"
Oct 30 05:30:01.078983 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:01.078417    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888970-qhwpt"
Oct 30 05:30:01.100196 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:01.097391    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ec92c7-29888970-g29zl"]
Oct 30 05:30:01.108963 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:01.108937    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8988\" (UniqueName: \"kubernetes.io/projected/b49de578-c4c0-4ae7-a090-4b9d8695b62c-kube-api-access-b8988\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888970-hsjd5\" (UID: \"b49de578-c4c0-4ae7-a090-4b9d8695b62c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888970-hsjd5"
Oct 30 05:30:01.111930 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:01.111855    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g22rz\" (UniqueName: \"kubernetes.io/projected/7dd199d3-31ed-41c5-8570-f6a1bc1cbdc9-kube-api-access-g22rz\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888970-k9zdq\" (UID: \"7dd199d3-31ed-41c5-8570-f6a1bc1cbdc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888970-k9zdq"
Oct 30 05:30:01.116707 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:01.116687    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-45607d-29888970-tzknr"]
Oct 30 05:30:01.116928 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:01.116912    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ec92c7-29888970-g29zl"
Oct 30 05:30:01.127453 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:01.127427    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888970-22ksr"]
Oct 30 05:30:01.127584 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:01.127477    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888970-hsjd5"]
Oct 30 05:30:01.127584 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:01.127503    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ec92c7-29888970-g29zl"]
Oct 30 05:30:01.127584 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:01.127530    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-45607d-29888970-tzknr"]
Oct 30 05:30:01.127757 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:01.127629    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-45607d-29888970-tzknr"
Oct 30 05:30:01.130803 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:01.130741    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxqhk\" (UniqueName: \"kubernetes.io/projected/c2b36ef9-dd65-43f2-88fe-f360f18ba0fc-kube-api-access-wxqhk\") pod \"maas-api-key-cleanup-e2e-worker-w3-889ca9-29888970-22ksr\" (UID: \"c2b36ef9-dd65-43f2-88fe-f360f18ba0fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888970-22ksr"
Oct 30 05:30:01.150356 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:01.143174    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czz75\" (UniqueName: \"kubernetes.io/projected/3ea1e4a0-ae9b-4a99-bf48-8f5a96e33e3b-kube-api-access-czz75\") pod \"maas-api-key-cleanup-e2e-worker-w0-b21507-29888970-5ssdn\" (UID: \"3ea1e4a0-ae9b-4a99-bf48-8f5a96e33e3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888970-5ssdn"
Oct 30 05:30:01.150356 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:01.143238    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ea1e4a0-ae9b-4a99-bf48-8f5a96e33e3b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b21507-29888970-5ssdn\" (UID: \"3ea1e4a0-ae9b-4a99-bf48-8f5a96e33e3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888970-5ssdn"
Oct 30 05:30:01.150356 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:01.145151    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ea1e4a0-ae9b-4a99-bf48-8f5a96e33e3b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b21507-29888970-5ssdn\" (UID: \"3ea1e4a0-ae9b-4a99-bf48-8f5a96e33e3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888970-5ssdn"
Oct 30 05:30:01.150356 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:01.147608    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888970-mf2ct"]
Oct 30 05:30:01.163773 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:01.163747    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888970-k9zdq"]
Oct 30 05:30:01.175318 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:01.175291    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888970-mf2ct"
Oct 30 05:30:01.274949 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:01.252114    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f696384-5b18-4ee5-8956-a4e09dfd54f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-45607d-29888970-tzknr\" (UID: \"0f696384-5b18-4ee5-8956-a4e09dfd54f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-45607d-29888970-tzknr"
Oct 30 05:30:01.274949 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:01.252494    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxfwf\" (UniqueName: \"kubernetes.io/projected/c687182c-2e85-4e99-8c2f-e8c972e37bcd-kube-api-access-kxfwf\") pod \"maas-api-key-cleanup-e2e-ipp-b-ec92c7-29888970-g29zl\" (UID: \"c687182c-2e85-4e99-8c2f-e8c972e37bcd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ec92c7-29888970-g29zl"
Oct 30 05:30:01.274949 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:01.252527    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c687182c-2e85-4e99-8c2f-e8c972e37bcd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-ec92c7-29888970-g29zl\" (UID: \"c687182c-2e85-4e99-8c2f-e8c972e37bcd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ec92c7-29888970-g29zl"
Oct 30 05:30:01.274949 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:01.252715    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8v6k\" (UniqueName: \"kubernetes.io/projected/0f696384-5b18-4ee5-8956-a4e09dfd54f2-kube-api-access-h8v6k\") pod \"maas-api-key-cleanup-e2e-ipp-b-45607d-29888970-tzknr\" (UID: \"0f696384-5b18-4ee5-8956-a4e09dfd54f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-45607d-29888970-tzknr"
Oct 30 05:30:01.280808 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:01.280781    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czz75\" (UniqueName: \"kubernetes.io/projected/3ea1e4a0-ae9b-4a99-bf48-8f5a96e33e3b-kube-api-access-czz75\") pod \"maas-api-key-cleanup-e2e-worker-w0-b21507-29888970-5ssdn\" (UID: \"3ea1e4a0-ae9b-4a99-bf48-8f5a96e33e3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888970-5ssdn"
Oct 30 05:30:01.517009 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:01.429959    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888970-k9zdq"
Oct 30 05:30:01.517009 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:01.430892    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxfwf\" (UniqueName: \"kubernetes.io/projected/c687182c-2e85-4e99-8c2f-e8c972e37bcd-kube-api-access-kxfwf\") pod \"maas-api-key-cleanup-e2e-ipp-b-ec92c7-29888970-g29zl\" (UID: \"c687182c-2e85-4e99-8c2f-e8c972e37bcd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ec92c7-29888970-g29zl"
Oct 30 05:30:01.517009 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:01.430955    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c687182c-2e85-4e99-8c2f-e8c972e37bcd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-ec92c7-29888970-g29zl\" (UID: \"c687182c-2e85-4e99-8c2f-e8c972e37bcd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ec92c7-29888970-g29zl"
Oct 30 05:30:01.517009 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:01.431085    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8v6k\" (UniqueName: \"kubernetes.io/projected/0f696384-5b18-4ee5-8956-a4e09dfd54f2-kube-api-access-h8v6k\") pod \"maas-api-key-cleanup-e2e-ipp-b-45607d-29888970-tzknr\" (UID: \"0f696384-5b18-4ee5-8956-a4e09dfd54f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-45607d-29888970-tzknr"
Oct 30 05:30:01.517009 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:01.431148    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f696384-5b18-4ee5-8956-a4e09dfd54f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-45607d-29888970-tzknr\" (UID: \"0f696384-5b18-4ee5-8956-a4e09dfd54f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-45607d-29888970-tzknr"
Oct 30 05:30:01.517009 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:01.454647    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888970-hsjd5"
Oct 30 05:30:01.517009 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:01.454918    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888970-22ksr"
Oct 30 05:30:01.517009 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:01.455745    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c687182c-2e85-4e99-8c2f-e8c972e37bcd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-ec92c7-29888970-g29zl\" (UID: \"c687182c-2e85-4e99-8c2f-e8c972e37bcd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ec92c7-29888970-g29zl"
Oct 30 05:30:01.517009 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:01.474770    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888970-5ssdn"]
Oct 30 05:30:01.574026 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:01.573640    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888970-5ssdn"
Oct 30 05:30:01.575092 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:01.575071    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8v6k\" (UniqueName: \"kubernetes.io/projected/0f696384-5b18-4ee5-8956-a4e09dfd54f2-kube-api-access-h8v6k\") pod \"maas-api-key-cleanup-e2e-ipp-b-45607d-29888970-tzknr\" (UID: \"0f696384-5b18-4ee5-8956-a4e09dfd54f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-45607d-29888970-tzknr"
Oct 30 05:30:01.732505 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:01.732464    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f696384-5b18-4ee5-8956-a4e09dfd54f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-45607d-29888970-tzknr\" (UID: \"0f696384-5b18-4ee5-8956-a4e09dfd54f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-45607d-29888970-tzknr"
Oct 30 05:30:01.752762 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:01.752290    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:01.791292 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:01.786217    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-45607d-29888970-tzknr"
Oct 30 05:30:01.860459 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:01.851734    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxfwf\" (UniqueName: \"kubernetes.io/projected/c687182c-2e85-4e99-8c2f-e8c972e37bcd-kube-api-access-kxfwf\") pod \"maas-api-key-cleanup-e2e-ipp-b-ec92c7-29888970-g29zl\" (UID: \"c687182c-2e85-4e99-8c2f-e8c972e37bcd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ec92c7-29888970-g29zl"
Oct 30 05:30:02.065173 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:02.064447    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ec92c7-29888970-g29zl"
Oct 30 05:30:02.558603 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:02.558571    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:02.668128 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:02.668038    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:02.682015 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:02.679954    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888970-22ksr"]
Oct 30 05:30:02.687138 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:30:02.687109    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d8a177e_9a55_45dc_8b18_548547e72b6b.slice/crio-5acae268574b2cde45a4d95a2f68694edb3217ecadf1077e5fdec6f2a39eb75c WatchSource:0}: Error finding container 5acae268574b2cde45a4d95a2f68694edb3217ecadf1077e5fdec6f2a39eb75c: Status 404 returned error can't find the container with id 5acae268574b2cde45a4d95a2f68694edb3217ecadf1077e5fdec6f2a39eb75c
Oct 30 05:30:02.694658 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:02.693991    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888970-k4phq"]
Oct 30 05:30:02.699819 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:02.699789    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888970-qhwpt"]
Oct 30 05:30:02.726171 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:02.726141    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888970-mf2ct"]
Oct 30 05:30:02.744396 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:30:02.744366    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf05b65d9_9992_445b_af08_28759a4ef2e2.slice/crio-53892149f69d7d70fa3472446e28fa13eb98bc76d914b4b11347a65043e62ea1 WatchSource:0}: Error finding container 53892149f69d7d70fa3472446e28fa13eb98bc76d914b4b11347a65043e62ea1: Status 404 returned error can't find the container with id 53892149f69d7d70fa3472446e28fa13eb98bc76d914b4b11347a65043e62ea1
Oct 30 05:30:02.790423 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:02.778366    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2f6fed-29888970-rvzpv"]
Oct 30 05:30:02.790423 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:30:02.780051    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbcc00ce_d37e_49fc_8755_7100bdfdd954.slice/crio-64357128e3b2deb9171e9582ac4dbf8ff45eecde584c061455cb8c1a43fffe63 WatchSource:0}: Error finding container 64357128e3b2deb9171e9582ac4dbf8ff45eecde584c061455cb8c1a43fffe63: Status 404 returned error can't find the container with id 64357128e3b2deb9171e9582ac4dbf8ff45eecde584c061455cb8c1a43fffe63
Oct 30 05:30:02.820544 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:02.820513    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-45607d-29888970-tzknr"]
Oct 30 05:30:03.261666 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:03.249836    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888970-5ssdn"]
Oct 30 05:30:03.261666 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:03.250287    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ec92c7-29888970-g29zl"]
Oct 30 05:30:03.261666 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:03.250336    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888970-k9zdq"]
Oct 30 05:30:03.265837 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:03.265794    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888970-hsjd5"]
Oct 30 05:30:03.596658 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:03.596620    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ec92c7-29888970-g29zl" event={"ID":"c687182c-2e85-4e99-8c2f-e8c972e37bcd","Type":"ContainerStarted","Data":"8cb8cc05d77f6f81b48aef868379828258f13fcc9c85d3977d3fe5e3baeec7f2"}
Oct 30 05:30:03.598694 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:03.598660    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888970-22ksr" event={"ID":"c2b36ef9-dd65-43f2-88fe-f360f18ba0fc","Type":"ContainerStarted","Data":"0ad0c64aa06605ff199f42b015517be7d1d790c694e7765c16ac14e159572dda"}
Oct 30 05:30:03.598847 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:03.598744    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888970-22ksr" event={"ID":"c2b36ef9-dd65-43f2-88fe-f360f18ba0fc","Type":"ContainerStarted","Data":"db7544bcf24fe8fffb15970d9b515d0e15eead05b26207751fe00d2414044cf3"}
Oct 30 05:30:03.600612 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:03.600585    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888970-k4phq" event={"ID":"2d8a177e-9a55-45dc-8b18-548547e72b6b","Type":"ContainerStarted","Data":"01c2b90dc75cff15742145c3eb2a51064b2bcac385cdc3510c642e8f2f740469"}
Oct 30 05:30:03.600690 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:03.600634    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888970-k4phq" event={"ID":"2d8a177e-9a55-45dc-8b18-548547e72b6b","Type":"ContainerStarted","Data":"5acae268574b2cde45a4d95a2f68694edb3217ecadf1077e5fdec6f2a39eb75c"}
Oct 30 05:30:03.604274 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:03.604245    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2f6fed-29888970-rvzpv" event={"ID":"fbcc00ce-d37e-49fc-8755-7100bdfdd954","Type":"ContainerStarted","Data":"64357128e3b2deb9171e9582ac4dbf8ff45eecde584c061455cb8c1a43fffe63"}
Oct 30 05:30:03.606019 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:03.605992    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888970-qhwpt" event={"ID":"08f30f4b-4298-494f-9fe3-de259a1991a0","Type":"ContainerStarted","Data":"4340094a7a27d241e8d6d101d6ac800d72c2e9892ac31e6c33d3cda2c27e10c3"}
Oct 30 05:30:03.606217 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:03.606190    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888970-qhwpt" event={"ID":"08f30f4b-4298-494f-9fe3-de259a1991a0","Type":"ContainerStarted","Data":"24e81cb4b62dc8ecfb8ea29617d49b7ba498dd3596108b1b3e110e2dede6b95a"}
Oct 30 05:30:03.608641 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:03.608617    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888970-5ssdn" event={"ID":"3ea1e4a0-ae9b-4a99-bf48-8f5a96e33e3b","Type":"ContainerStarted","Data":"4867542f1fd5f406c92eee5275e8942ba6a952cdc823150c72faa47eb9d64617"}
Oct 30 05:30:03.609290 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:03.609228    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:03.610409 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:03.610382    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888970-mf2ct" event={"ID":"f05b65d9-9992-445b-af08-28759a4ef2e2","Type":"ContainerStarted","Data":"92671f7dd1a13c9b8f9ca9d032670318ca188f9488f1db79e35fb6bc9293df7d"}
Oct 30 05:30:03.610522 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:03.610433    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888970-mf2ct" event={"ID":"f05b65d9-9992-445b-af08-28759a4ef2e2","Type":"ContainerStarted","Data":"53892149f69d7d70fa3472446e28fa13eb98bc76d914b4b11347a65043e62ea1"}
Oct 30 05:30:03.611736 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:03.611717    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888970-k9zdq" event={"ID":"7dd199d3-31ed-41c5-8570-f6a1bc1cbdc9","Type":"ContainerStarted","Data":"bae0890e2acc0f8571db92ccbf7749bfa09cadc7aa6eb8f815d8611f032fa886"}
Oct 30 05:30:03.613160 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:03.613136    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888970-hsjd5" event={"ID":"b49de578-c4c0-4ae7-a090-4b9d8695b62c","Type":"ContainerStarted","Data":"744486f17b1ccacd3e6c3a750b70686e57d5d1fed961401cedca189646b53394"}
Oct 30 05:30:03.614485 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:03.614462    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-45607d-29888970-tzknr" event={"ID":"0f696384-5b18-4ee5-8956-a4e09dfd54f2","Type":"ContainerStarted","Data":"f2540c055717613d8661afea8cde03407c3f0309d37964f6f22e79178e4a3d01"}
Oct 30 05:30:03.757014 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:03.755649    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888970-k4phq" podStartSLOduration=3.755631198 podStartE2EDuration="3.755631198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:03.742417408 +0000 UTC m=+2732.966237314" watchObservedRunningTime="2026-10-30 05:30:03.755631198 +0000 UTC m=+2732.979451100"
Oct 30 05:30:03.757014 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:03.755757    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888970-22ksr" podStartSLOduration=3.7557512749999997 podStartE2EDuration="3.755751275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:03.698353829 +0000 UTC m=+2732.922173748" watchObservedRunningTime="2026-10-30 05:30:03.755751275 +0000 UTC m=+2732.979571177"
Oct 30 05:30:03.791151 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:03.789168    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888970-mf2ct" podStartSLOduration=3.789134684 podStartE2EDuration="3.789134684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:03.781089543 +0000 UTC m=+2733.004909464" watchObservedRunningTime="2026-10-30 05:30:03.789134684 +0000 UTC m=+2733.012954586"
Oct 30 05:30:03.826167 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:03.826107    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888970-qhwpt" podStartSLOduration=3.82608955 podStartE2EDuration="3.82608955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:03.822786895 +0000 UTC m=+2733.046606796" watchObservedRunningTime="2026-10-30 05:30:03.82608955 +0000 UTC m=+2733.049909453"
Oct 30 05:30:04.077781 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.077751    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt"]
Oct 30 05:30:04.083523 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.083501    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt"
Oct 30 05:30:04.115279 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.108615    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt"]
Oct 30 05:30:04.115580 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.115365    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9"]
Oct 30 05:30:04.120668 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.120646    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9"
Oct 30 05:30:04.143387 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.142947    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9"]
Oct 30 05:30:04.252363 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.252324    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ef747b3-e680-47d5-8849-e26bd0946d15-istio-data\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt\" (UID: \"3ef747b3-e680-47d5-8849-e26bd0946d15\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt"
Oct 30 05:30:04.252536 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.252383    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1ddb31-cd93-4f89-9968-e94fd4484b6f-credential-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9\" (UID: \"ee1ddb31-cd93-4f89-9968-e94fd4484b6f\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9"
Oct 30 05:30:04.252536 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.252472    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee1ddb31-cd93-4f89-9968-e94fd4484b6f-istio-token\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9\" (UID: \"ee1ddb31-cd93-4f89-9968-e94fd4484b6f\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9"
Oct 30 05:30:04.252536 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.252524    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ef747b3-e680-47d5-8849-e26bd0946d15-workload-certs\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt\" (UID: \"3ef747b3-e680-47d5-8849-e26bd0946d15\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt"
Oct 30 05:30:04.252646 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.252578    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee1ddb31-cd93-4f89-9968-e94fd4484b6f-istio-data\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9\" (UID: \"ee1ddb31-cd93-4f89-9968-e94fd4484b6f\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9"
Oct 30 05:30:04.252646 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.252630    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ef747b3-e680-47d5-8849-e26bd0946d15-istio-podinfo\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt\" (UID: \"3ef747b3-e680-47d5-8849-e26bd0946d15\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt"
Oct 30 05:30:04.252712 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.252669    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ef747b3-e680-47d5-8849-e26bd0946d15-istio-envoy\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt\" (UID: \"3ef747b3-e680-47d5-8849-e26bd0946d15\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt"
Oct 30 05:30:04.252764 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.252749    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee1ddb31-cd93-4f89-9968-e94fd4484b6f-workload-certs\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9\" (UID: \"ee1ddb31-cd93-4f89-9968-e94fd4484b6f\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9"
Oct 30 05:30:04.252811 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.252778    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcknz\" (UniqueName: \"kubernetes.io/projected/3ef747b3-e680-47d5-8849-e26bd0946d15-kube-api-access-jcknz\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt\" (UID: \"3ef747b3-e680-47d5-8849-e26bd0946d15\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt"
Oct 30 05:30:04.252849 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.252813    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ef747b3-e680-47d5-8849-e26bd0946d15-credential-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt\" (UID: \"3ef747b3-e680-47d5-8849-e26bd0946d15\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt"
Oct 30 05:30:04.252935 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.252917    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1ddb31-cd93-4f89-9968-e94fd4484b6f-workload-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9\" (UID: \"ee1ddb31-cd93-4f89-9968-e94fd4484b6f\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9"
Oct 30 05:30:04.253003 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.252989    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee1ddb31-cd93-4f89-9968-e94fd4484b6f-istiod-ca-cert\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9\" (UID: \"ee1ddb31-cd93-4f89-9968-e94fd4484b6f\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9"
Oct 30 05:30:04.253041 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.253015    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee1ddb31-cd93-4f89-9968-e94fd4484b6f-istio-envoy\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9\" (UID: \"ee1ddb31-cd93-4f89-9968-e94fd4484b6f\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9"
Oct 30 05:30:04.253111 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.253092    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ef747b3-e680-47d5-8849-e26bd0946d15-istiod-ca-cert\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt\" (UID: \"3ef747b3-e680-47d5-8849-e26bd0946d15\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt"
Oct 30 05:30:04.253150 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.253139    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee1ddb31-cd93-4f89-9968-e94fd4484b6f-istio-podinfo\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9\" (UID: \"ee1ddb31-cd93-4f89-9968-e94fd4484b6f\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9"
Oct 30 05:30:04.253183 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.253158    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ef747b3-e680-47d5-8849-e26bd0946d15-istio-token\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt\" (UID: \"3ef747b3-e680-47d5-8849-e26bd0946d15\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt"
Oct 30 05:30:04.253223 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.253208    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9qnd\" (UniqueName: \"kubernetes.io/projected/ee1ddb31-cd93-4f89-9968-e94fd4484b6f-kube-api-access-z9qnd\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9\" (UID: \"ee1ddb31-cd93-4f89-9968-e94fd4484b6f\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9"
Oct 30 05:30:04.253280 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.253269    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ef747b3-e680-47d5-8849-e26bd0946d15-workload-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt\" (UID: \"3ef747b3-e680-47d5-8849-e26bd0946d15\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt"
Oct 30 05:30:04.354553 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.354514    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1ddb31-cd93-4f89-9968-e94fd4484b6f-workload-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9\" (UID: \"ee1ddb31-cd93-4f89-9968-e94fd4484b6f\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9"
Oct 30 05:30:04.354763 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.354591    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee1ddb31-cd93-4f89-9968-e94fd4484b6f-istiod-ca-cert\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9\" (UID: \"ee1ddb31-cd93-4f89-9968-e94fd4484b6f\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9"
Oct 30 05:30:04.354763 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.354613    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee1ddb31-cd93-4f89-9968-e94fd4484b6f-istio-envoy\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9\" (UID: \"ee1ddb31-cd93-4f89-9968-e94fd4484b6f\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9"
Oct 30 05:30:04.354763 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.354678    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ef747b3-e680-47d5-8849-e26bd0946d15-istiod-ca-cert\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt\" (UID: \"3ef747b3-e680-47d5-8849-e26bd0946d15\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt"
Oct 30 05:30:04.354763 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.354722    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee1ddb31-cd93-4f89-9968-e94fd4484b6f-istio-podinfo\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9\" (UID: \"ee1ddb31-cd93-4f89-9968-e94fd4484b6f\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9"
Oct 30 05:30:04.354763 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.354748    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ef747b3-e680-47d5-8849-e26bd0946d15-istio-token\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt\" (UID: \"3ef747b3-e680-47d5-8849-e26bd0946d15\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt"
Oct 30 05:30:04.355082 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.354790    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9qnd\" (UniqueName: \"kubernetes.io/projected/ee1ddb31-cd93-4f89-9968-e94fd4484b6f-kube-api-access-z9qnd\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9\" (UID: \"ee1ddb31-cd93-4f89-9968-e94fd4484b6f\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9"
Oct 30 05:30:04.355082 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.354832    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ef747b3-e680-47d5-8849-e26bd0946d15-workload-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt\" (UID: \"3ef747b3-e680-47d5-8849-e26bd0946d15\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt"
Oct 30 05:30:04.355082 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.354892    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ef747b3-e680-47d5-8849-e26bd0946d15-istio-data\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt\" (UID: \"3ef747b3-e680-47d5-8849-e26bd0946d15\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt"
Oct 30 05:30:04.355082 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.354947    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1ddb31-cd93-4f89-9968-e94fd4484b6f-credential-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9\" (UID: \"ee1ddb31-cd93-4f89-9968-e94fd4484b6f\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9"
Oct 30 05:30:04.355082 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.354983    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1ddb31-cd93-4f89-9968-e94fd4484b6f-workload-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9\" (UID: \"ee1ddb31-cd93-4f89-9968-e94fd4484b6f\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9"
Oct 30 05:30:04.355082 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.355005    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee1ddb31-cd93-4f89-9968-e94fd4484b6f-istio-token\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9\" (UID: \"ee1ddb31-cd93-4f89-9968-e94fd4484b6f\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9"
Oct 30 05:30:04.355082 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.355071    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ef747b3-e680-47d5-8849-e26bd0946d15-workload-certs\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt\" (UID: \"3ef747b3-e680-47d5-8849-e26bd0946d15\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt"
Oct 30 05:30:04.355432 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.355095    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee1ddb31-cd93-4f89-9968-e94fd4484b6f-istio-data\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9\" (UID: \"ee1ddb31-cd93-4f89-9968-e94fd4484b6f\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9"
Oct 30 05:30:04.355432 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.355144    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ef747b3-e680-47d5-8849-e26bd0946d15-istio-podinfo\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt\" (UID: \"3ef747b3-e680-47d5-8849-e26bd0946d15\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt"
Oct 30 05:30:04.355432 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.355182    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ef747b3-e680-47d5-8849-e26bd0946d15-istio-envoy\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt\" (UID: \"3ef747b3-e680-47d5-8849-e26bd0946d15\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt"
Oct 30 05:30:04.355432 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.355282    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee1ddb31-cd93-4f89-9968-e94fd4484b6f-workload-certs\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9\" (UID: \"ee1ddb31-cd93-4f89-9968-e94fd4484b6f\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9"
Oct 30 05:30:04.355432 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.355289    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ef747b3-e680-47d5-8849-e26bd0946d15-istio-data\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt\" (UID: \"3ef747b3-e680-47d5-8849-e26bd0946d15\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt"
Oct 30 05:30:04.355432 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.355303    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcknz\" (UniqueName: \"kubernetes.io/projected/3ef747b3-e680-47d5-8849-e26bd0946d15-kube-api-access-jcknz\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt\" (UID: \"3ef747b3-e680-47d5-8849-e26bd0946d15\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt"
Oct 30 05:30:04.355432 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.355356    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ef747b3-e680-47d5-8849-e26bd0946d15-credential-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt\" (UID: \"3ef747b3-e680-47d5-8849-e26bd0946d15\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt"
Oct 30 05:30:04.355784 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.355586    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ef747b3-e680-47d5-8849-e26bd0946d15-istiod-ca-cert\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt\" (UID: \"3ef747b3-e680-47d5-8849-e26bd0946d15\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt"
Oct 30 05:30:04.355784 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.355722    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ef747b3-e680-47d5-8849-e26bd0946d15-credential-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt\" (UID: \"3ef747b3-e680-47d5-8849-e26bd0946d15\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt"
Oct 30 05:30:04.355913 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.355820    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1ddb31-cd93-4f89-9968-e94fd4484b6f-credential-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9\" (UID: \"ee1ddb31-cd93-4f89-9968-e94fd4484b6f\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9"
Oct 30 05:30:04.356047 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.356027    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ef747b3-e680-47d5-8849-e26bd0946d15-workload-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt\" (UID: \"3ef747b3-e680-47d5-8849-e26bd0946d15\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt"
Oct 30 05:30:04.356197 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.356159    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ef747b3-e680-47d5-8849-e26bd0946d15-workload-certs\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt\" (UID: \"3ef747b3-e680-47d5-8849-e26bd0946d15\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt"
Oct 30 05:30:04.356572 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.356283    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee1ddb31-cd93-4f89-9968-e94fd4484b6f-workload-certs\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9\" (UID: \"ee1ddb31-cd93-4f89-9968-e94fd4484b6f\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9"
Oct 30 05:30:04.356572 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.356449    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee1ddb31-cd93-4f89-9968-e94fd4484b6f-istio-data\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9\" (UID: \"ee1ddb31-cd93-4f89-9968-e94fd4484b6f\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9"
Oct 30 05:30:04.357161 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.357135    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee1ddb31-cd93-4f89-9968-e94fd4484b6f-istiod-ca-cert\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9\" (UID: \"ee1ddb31-cd93-4f89-9968-e94fd4484b6f\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9"
Oct 30 05:30:04.358083 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.357825    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee1ddb31-cd93-4f89-9968-e94fd4484b6f-istio-envoy\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9\" (UID: \"ee1ddb31-cd93-4f89-9968-e94fd4484b6f\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9"
Oct 30 05:30:04.358787 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.358768    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ef747b3-e680-47d5-8849-e26bd0946d15-istio-envoy\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt\" (UID: \"3ef747b3-e680-47d5-8849-e26bd0946d15\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt"
Oct 30 05:30:04.358893 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.358872    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ef747b3-e680-47d5-8849-e26bd0946d15-istio-podinfo\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt\" (UID: \"3ef747b3-e680-47d5-8849-e26bd0946d15\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt"
Oct 30 05:30:04.358995 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.358929    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee1ddb31-cd93-4f89-9968-e94fd4484b6f-istio-podinfo\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9\" (UID: \"ee1ddb31-cd93-4f89-9968-e94fd4484b6f\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9"
Oct 30 05:30:04.383550 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.382766    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee1ddb31-cd93-4f89-9968-e94fd4484b6f-istio-token\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9\" (UID: \"ee1ddb31-cd93-4f89-9968-e94fd4484b6f\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9"
Oct 30 05:30:04.385931 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.385352    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9qnd\" (UniqueName: \"kubernetes.io/projected/ee1ddb31-cd93-4f89-9968-e94fd4484b6f-kube-api-access-z9qnd\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9\" (UID: \"ee1ddb31-cd93-4f89-9968-e94fd4484b6f\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9"
Oct 30 05:30:04.387688 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.387656    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ef747b3-e680-47d5-8849-e26bd0946d15-istio-token\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt\" (UID: \"3ef747b3-e680-47d5-8849-e26bd0946d15\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt"
Oct 30 05:30:04.388150 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.388102    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcknz\" (UniqueName: \"kubernetes.io/projected/3ef747b3-e680-47d5-8849-e26bd0946d15-kube-api-access-jcknz\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt\" (UID: \"3ef747b3-e680-47d5-8849-e26bd0946d15\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt"
Oct 30 05:30:04.498826 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.498801    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt"
Oct 30 05:30:04.505616 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.505589    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9"
Oct 30 05:30:04.627916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.627862    5898 generic.go:356] "Generic (PLEG): container finished" podID="c687182c-2e85-4e99-8c2f-e8c972e37bcd" containerID="71a9307145839ab190ee23a55cf67b47c11fa4270cae5de6493a7c339e132c8a" exitCode=60
Oct 30 05:30:04.628343 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.627968    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ec92c7-29888970-g29zl" event={"ID":"c687182c-2e85-4e99-8c2f-e8c972e37bcd","Type":"ContainerDied","Data":"71a9307145839ab190ee23a55cf67b47c11fa4270cae5de6493a7c339e132c8a"}
Oct 30 05:30:04.628428 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.628337    5898 scope.go:122] "RemoveContainer" containerID="71a9307145839ab190ee23a55cf67b47c11fa4270cae5de6493a7c339e132c8a"
Oct 30 05:30:04.631176 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.631148    5898 generic.go:356] "Generic (PLEG): container finished" podID="c2b36ef9-dd65-43f2-88fe-f360f18ba0fc" containerID="0ad0c64aa06605ff199f42b015517be7d1d790c694e7765c16ac14e159572dda" exitCode=60
Oct 30 05:30:04.631319 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.631222    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888970-22ksr" event={"ID":"c2b36ef9-dd65-43f2-88fe-f360f18ba0fc","Type":"ContainerDied","Data":"0ad0c64aa06605ff199f42b015517be7d1d790c694e7765c16ac14e159572dda"}
Oct 30 05:30:04.631628 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.631606    5898 scope.go:122] "RemoveContainer" containerID="0ad0c64aa06605ff199f42b015517be7d1d790c694e7765c16ac14e159572dda"
Oct 30 05:30:04.634373 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.634256    5898 generic.go:356] "Generic (PLEG): container finished" podID="2d8a177e-9a55-45dc-8b18-548547e72b6b" containerID="01c2b90dc75cff15742145c3eb2a51064b2bcac385cdc3510c642e8f2f740469" exitCode=60
Oct 30 05:30:04.634373 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.634329    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888970-k4phq" event={"ID":"2d8a177e-9a55-45dc-8b18-548547e72b6b","Type":"ContainerDied","Data":"01c2b90dc75cff15742145c3eb2a51064b2bcac385cdc3510c642e8f2f740469"}
Oct 30 05:30:04.634703 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.634681    5898 scope.go:122] "RemoveContainer" containerID="01c2b90dc75cff15742145c3eb2a51064b2bcac385cdc3510c642e8f2f740469"
Oct 30 05:30:04.637520 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.637193    5898 generic.go:356] "Generic (PLEG): container finished" podID="fbcc00ce-d37e-49fc-8755-7100bdfdd954" containerID="de96e3d9a4ad447f7985abf8ca2e7a39b54b15f2c41225618d9457ee9d8b8aa1" exitCode=60
Oct 30 05:30:04.637520 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.637470    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2f6fed-29888970-rvzpv" event={"ID":"fbcc00ce-d37e-49fc-8755-7100bdfdd954","Type":"ContainerDied","Data":"de96e3d9a4ad447f7985abf8ca2e7a39b54b15f2c41225618d9457ee9d8b8aa1"}
Oct 30 05:30:04.638158 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.637963    5898 scope.go:122] "RemoveContainer" containerID="de96e3d9a4ad447f7985abf8ca2e7a39b54b15f2c41225618d9457ee9d8b8aa1"
Oct 30 05:30:04.639339 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.639313    5898 generic.go:356] "Generic (PLEG): container finished" podID="08f30f4b-4298-494f-9fe3-de259a1991a0" containerID="4340094a7a27d241e8d6d101d6ac800d72c2e9892ac31e6c33d3cda2c27e10c3" exitCode=60
Oct 30 05:30:04.639434 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.639395    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888970-qhwpt" event={"ID":"08f30f4b-4298-494f-9fe3-de259a1991a0","Type":"ContainerDied","Data":"4340094a7a27d241e8d6d101d6ac800d72c2e9892ac31e6c33d3cda2c27e10c3"}
Oct 30 05:30:04.639832 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.639753    5898 scope.go:122] "RemoveContainer" containerID="4340094a7a27d241e8d6d101d6ac800d72c2e9892ac31e6c33d3cda2c27e10c3"
Oct 30 05:30:04.647958 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.646112    5898 generic.go:356] "Generic (PLEG): container finished" podID="3ea1e4a0-ae9b-4a99-bf48-8f5a96e33e3b" containerID="896764cc4d48cb7d7cc99c17b1cadf04bf9c635fa355a3aa1e3c73376fb9cb8e" exitCode=60
Oct 30 05:30:04.647958 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.646487    5898 scope.go:122] "RemoveContainer" containerID="896764cc4d48cb7d7cc99c17b1cadf04bf9c635fa355a3aa1e3c73376fb9cb8e"
Oct 30 05:30:04.647958 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.646518    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888970-5ssdn" event={"ID":"3ea1e4a0-ae9b-4a99-bf48-8f5a96e33e3b","Type":"ContainerDied","Data":"896764cc4d48cb7d7cc99c17b1cadf04bf9c635fa355a3aa1e3c73376fb9cb8e"}
Oct 30 05:30:04.653382 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.653063    5898 generic.go:356] "Generic (PLEG): container finished" podID="f05b65d9-9992-445b-af08-28759a4ef2e2" containerID="92671f7dd1a13c9b8f9ca9d032670318ca188f9488f1db79e35fb6bc9293df7d" exitCode=60
Oct 30 05:30:04.653382 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.653147    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888970-mf2ct" event={"ID":"f05b65d9-9992-445b-af08-28759a4ef2e2","Type":"ContainerDied","Data":"92671f7dd1a13c9b8f9ca9d032670318ca188f9488f1db79e35fb6bc9293df7d"}
Oct 30 05:30:04.653382 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.653330    5898 scope.go:122] "RemoveContainer" containerID="92671f7dd1a13c9b8f9ca9d032670318ca188f9488f1db79e35fb6bc9293df7d"
Oct 30 05:30:04.656502 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.656484    5898 generic.go:356] "Generic (PLEG): container finished" podID="7dd199d3-31ed-41c5-8570-f6a1bc1cbdc9" containerID="0fd631738b05bedd5d567ab46d874b964cb344c873053302ca3e4d3418580400" exitCode=60
Oct 30 05:30:04.656602 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.656505    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888970-k9zdq" event={"ID":"7dd199d3-31ed-41c5-8570-f6a1bc1cbdc9","Type":"ContainerDied","Data":"0fd631738b05bedd5d567ab46d874b964cb344c873053302ca3e4d3418580400"}
Oct 30 05:30:04.656819 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.656800    5898 scope.go:122] "RemoveContainer" containerID="0fd631738b05bedd5d567ab46d874b964cb344c873053302ca3e4d3418580400"
Oct 30 05:30:04.658408 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.658384    5898 generic.go:356] "Generic (PLEG): container finished" podID="b49de578-c4c0-4ae7-a090-4b9d8695b62c" containerID="6348f75d59a8c8b5df4baacb9fcc84d4441d8c1a92df9ea686e1d7112c2c9e15" exitCode=60
Oct 30 05:30:04.658508 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.658472    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888970-hsjd5" event={"ID":"b49de578-c4c0-4ae7-a090-4b9d8695b62c","Type":"ContainerDied","Data":"6348f75d59a8c8b5df4baacb9fcc84d4441d8c1a92df9ea686e1d7112c2c9e15"}
Oct 30 05:30:04.658679 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.658661    5898 scope.go:122] "RemoveContainer" containerID="6348f75d59a8c8b5df4baacb9fcc84d4441d8c1a92df9ea686e1d7112c2c9e15"
Oct 30 05:30:04.661121 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.661090    5898 generic.go:356] "Generic (PLEG): container finished" podID="0f696384-5b18-4ee5-8956-a4e09dfd54f2" containerID="b1cfc72dbae0493f29503200dbeb7653e1f7ee7fe2451bed19936686cc951b6d" exitCode=60
Oct 30 05:30:04.661261 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.661154    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-45607d-29888970-tzknr" event={"ID":"0f696384-5b18-4ee5-8956-a4e09dfd54f2","Type":"ContainerDied","Data":"b1cfc72dbae0493f29503200dbeb7653e1f7ee7fe2451bed19936686cc951b6d"}
Oct 30 05:30:04.661472 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.661451    5898 scope.go:122] "RemoveContainer" containerID="b1cfc72dbae0493f29503200dbeb7653e1f7ee7fe2451bed19936686cc951b6d"
Oct 30 05:30:04.848431 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.848406    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt"]
Oct 30 05:30:04.863438 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.863410    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9"]
Oct 30 05:30:04.889648 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:30:04.889605    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ef747b3_e680_47d5_8849_e26bd0946d15.slice/crio-d718c606212d54458a12d09c696f8778fe3e866b14f8db9a4b7741a5ec83f19f WatchSource:0}: Error finding container d718c606212d54458a12d09c696f8778fe3e866b14f8db9a4b7741a5ec83f19f: Status 404 returned error can't find the container with id d718c606212d54458a12d09c696f8778fe3e866b14f8db9a4b7741a5ec83f19f
Oct 30 05:30:04.890220 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:30:04.890180    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee1ddb31_cd93_4f89_9968_e94fd4484b6f.slice/crio-ecc704c72f08a62854b67c91f9171645a27f8f9fe1a27ae7c61b1d2dd03f3670 WatchSource:0}: Error finding container ecc704c72f08a62854b67c91f9171645a27f8f9fe1a27ae7c61b1d2dd03f3670: Status 404 returned error can't find the container with id ecc704c72f08a62854b67c91f9171645a27f8f9fe1a27ae7c61b1d2dd03f3670
Oct 30 05:30:04.893857 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.893825    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:30:04.894002 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.893912    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:30:04.894002 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.893942    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:30:04.898385 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.898281    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:30:04.898385 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.898371    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:30:04.898602 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:04.898419    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:30:05.655941 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:05.646441    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:05.758405 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:05.749776    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888970-hsjd5" event={"ID":"b49de578-c4c0-4ae7-a090-4b9d8695b62c","Type":"ContainerStarted","Data":"8f1d69ea8b7eb3bb7bb783daddb1907eb1df860a0171c7ba28890c649d05756d"}
Oct 30 05:30:05.798709 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:05.794380    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-45607d-29888970-tzknr" event={"ID":"0f696384-5b18-4ee5-8956-a4e09dfd54f2","Type":"ContainerStarted","Data":"ad5868221654878de12844f409b65f839c2e9e4d39daa39a679239386250f654"}
Oct 30 05:30:05.827925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:05.820300    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888970-hsjd5" podStartSLOduration=5.820278388 podStartE2EDuration="5.820278388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:05.798752337 +0000 UTC m=+2735.022572255" watchObservedRunningTime="2026-10-30 05:30:05.820278388 +0000 UTC m=+2735.044098290"
Oct 30 05:30:05.863477 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:05.862338    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt" event={"ID":"3ef747b3-e680-47d5-8849-e26bd0946d15","Type":"ContainerStarted","Data":"ca11db02b22d6220f257314ecf5cecc9edbc75686993f7d086b77415449b07e0"}
Oct 30 05:30:05.863477 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:05.862404    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt" event={"ID":"3ef747b3-e680-47d5-8849-e26bd0946d15","Type":"ContainerStarted","Data":"d718c606212d54458a12d09c696f8778fe3e866b14f8db9a4b7741a5ec83f19f"}
Oct 30 05:30:05.866285 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:05.866250    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ec92c7-29888970-g29zl" event={"ID":"c687182c-2e85-4e99-8c2f-e8c972e37bcd","Type":"ContainerStarted","Data":"2ac71fdcceff1741a972fa94a0bf4950bb895e0129270d54d515072964077648"}
Oct 30 05:30:05.868551 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:05.868528    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9" event={"ID":"ee1ddb31-cd93-4f89-9968-e94fd4484b6f","Type":"ContainerStarted","Data":"d607dfc259e9b9ba4df94affb54aaa3bfd37cfabbf75a7090b4ff7b6f9f03029"}
Oct 30 05:30:05.868680 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:05.868572    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9" event={"ID":"ee1ddb31-cd93-4f89-9968-e94fd4484b6f","Type":"ContainerStarted","Data":"ecc704c72f08a62854b67c91f9171645a27f8f9fe1a27ae7c61b1d2dd03f3670"}
Oct 30 05:30:05.870578 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:05.870554    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888970-22ksr" event={"ID":"c2b36ef9-dd65-43f2-88fe-f360f18ba0fc","Type":"ContainerStarted","Data":"aae13aea4ae59b1f5b10be075615b066647696708c6b4e88801fc4efd58b000e"}
Oct 30 05:30:05.872005 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:05.871982    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888970-k4phq" event={"ID":"2d8a177e-9a55-45dc-8b18-548547e72b6b","Type":"ContainerStarted","Data":"601bf217659d0ddee8825450f2851d9dd43ad4d7defe02603c0112348ccc8510"}
Oct 30 05:30:05.873856 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:05.873836    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2f6fed-29888970-rvzpv" event={"ID":"fbcc00ce-d37e-49fc-8755-7100bdfdd954","Type":"ContainerStarted","Data":"2522a38c4e5cd85df5de546edc8a2ffb7deaa56625572636a2aa48cd5e0e986f"}
Oct 30 05:30:05.901592 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:05.890291    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-45607d-29888970-tzknr" podStartSLOduration=5.890271491 podStartE2EDuration="5.890271491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:05.888589876 +0000 UTC m=+2735.112409800" watchObservedRunningTime="2026-10-30 05:30:05.890271491 +0000 UTC m=+2735.114091396"
Oct 30 05:30:05.960626 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:05.959689    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888970-qhwpt" event={"ID":"08f30f4b-4298-494f-9fe3-de259a1991a0","Type":"ContainerStarted","Data":"5d356f0927897b56a3d3bb8d95a657b95ca3e3c673936fccae40deb8c022e57b"}
Oct 30 05:30:05.963610 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:05.963585    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888970-5ssdn" event={"ID":"3ea1e4a0-ae9b-4a99-bf48-8f5a96e33e3b","Type":"ContainerStarted","Data":"924898dc6a58c15b456b721a72ae03bdddce9123e2c8874c10f8215b5b508a11"}
Oct 30 05:30:05.963977 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:05.963958    5898 scope.go:122] "RemoveContainer" containerID="924898dc6a58c15b456b721a72ae03bdddce9123e2c8874c10f8215b5b508a11"
Oct 30 05:30:05.964314 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:30:05.964294    5898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b21507-29888970-5ssdn_odh-ai-gateway-infra(3ea1e4a0-ae9b-4a99-bf48-8f5a96e33e3b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888970-5ssdn" podUID="3ea1e4a0-ae9b-4a99-bf48-8f5a96e33e3b"
Oct 30 05:30:05.965832 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:05.965786    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt" podStartSLOduration=1.965770426 podStartE2EDuration="1.965770426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:05.961395108 +0000 UTC m=+2735.185215038" watchObservedRunningTime="2026-10-30 05:30:05.965770426 +0000 UTC m=+2735.189590328"
Oct 30 05:30:05.966841 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:05.966063    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888970-mf2ct" event={"ID":"f05b65d9-9992-445b-af08-28759a4ef2e2","Type":"ContainerStarted","Data":"e32f92fea09a45031c6a231231adc04ac51a8c240003107073418609dbe35290"}
Oct 30 05:30:05.966841 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:05.966411    5898 scope.go:122] "RemoveContainer" containerID="e32f92fea09a45031c6a231231adc04ac51a8c240003107073418609dbe35290"
Oct 30 05:30:05.966841 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:30:05.966793    5898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888970-mf2ct_odh-ai-gateway-infra(f05b65d9-9992-445b-af08-28759a4ef2e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888970-mf2ct" podUID="f05b65d9-9992-445b-af08-28759a4ef2e2"
Oct 30 05:30:05.970615 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:05.970592    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888970-k9zdq" event={"ID":"7dd199d3-31ed-41c5-8570-f6a1bc1cbdc9","Type":"ContainerStarted","Data":"7f6ad6199aeae680ad3e8880e44aee1ad1292a5d966b31111c773c20ca9ba026"}
Oct 30 05:30:06.129422 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:06.129359    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ec92c7-29888970-g29zl" podStartSLOduration=6.129339238 podStartE2EDuration="6.129339238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:06.088032471 +0000 UTC m=+2735.311852373" watchObservedRunningTime="2026-10-30 05:30:06.129339238 +0000 UTC m=+2735.353159141"
Oct 30 05:30:06.263819 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:06.263765    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9" podStartSLOduration=2.263750804 podStartE2EDuration="2.263750804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:06.215059914 +0000 UTC m=+2735.438879852" watchObservedRunningTime="2026-10-30 05:30:06.263750804 +0000 UTC m=+2735.487570759"
Oct 30 05:30:06.315933 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:06.314239    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888970-k9zdq" podStartSLOduration=6.314217622 podStartE2EDuration="6.314217622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:06.302104525 +0000 UTC m=+2735.525924449" watchObservedRunningTime="2026-10-30 05:30:06.314217622 +0000 UTC m=+2735.538037524"
Oct 30 05:30:06.413927 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:06.391244    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2f6fed-29888970-rvzpv" podStartSLOduration=6.391225066 podStartE2EDuration="6.391225066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:06.39116607 +0000 UTC m=+2735.614985971" watchObservedRunningTime="2026-10-30 05:30:06.391225066 +0000 UTC m=+2735.615044968"
Oct 30 05:30:06.501933 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:06.501807    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt"
Oct 30 05:30:06.502263 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:06.502237    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Oct 30 05:30:06.502446 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:06.502421    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt" podUID="3ef747b3-e680-47d5-8849-e26bd0946d15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Oct 30 05:30:06.507453 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:06.507431    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9"
Oct 30 05:30:06.513037 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:06.507743    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Oct 30 05:30:06.513248 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:06.513220    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9" podUID="ee1ddb31-cd93-4f89-9968-e94fd4484b6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Oct 30 05:30:07.033657 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:07.033622    5898 generic.go:356] "Generic (PLEG): container finished" podID="7dd199d3-31ed-41c5-8570-f6a1bc1cbdc9" containerID="7f6ad6199aeae680ad3e8880e44aee1ad1292a5d966b31111c773c20ca9ba026" exitCode=60
Oct 30 05:30:07.034193 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:07.033763    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888970-k9zdq" event={"ID":"7dd199d3-31ed-41c5-8570-f6a1bc1cbdc9","Type":"ContainerDied","Data":"7f6ad6199aeae680ad3e8880e44aee1ad1292a5d966b31111c773c20ca9ba026"}
Oct 30 05:30:07.034193 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:07.033824    5898 scope.go:122] "RemoveContainer" containerID="0fd631738b05bedd5d567ab46d874b964cb344c873053302ca3e4d3418580400"
Oct 30 05:30:07.034193 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:07.034053    5898 scope.go:122] "RemoveContainer" containerID="7f6ad6199aeae680ad3e8880e44aee1ad1292a5d966b31111c773c20ca9ba026"
Oct 30 05:30:07.035037 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:30:07.034527    5898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888970-k9zdq_odh-ai-gateway-infra(7dd199d3-31ed-41c5-8570-f6a1bc1cbdc9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888970-k9zdq" podUID="7dd199d3-31ed-41c5-8570-f6a1bc1cbdc9"
Oct 30 05:30:07.038254 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:07.037670    5898 generic.go:356] "Generic (PLEG): container finished" podID="b49de578-c4c0-4ae7-a090-4b9d8695b62c" containerID="8f1d69ea8b7eb3bb7bb783daddb1907eb1df860a0171c7ba28890c649d05756d" exitCode=60
Oct 30 05:30:07.038254 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:07.037796    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888970-hsjd5" event={"ID":"b49de578-c4c0-4ae7-a090-4b9d8695b62c","Type":"ContainerDied","Data":"8f1d69ea8b7eb3bb7bb783daddb1907eb1df860a0171c7ba28890c649d05756d"}
Oct 30 05:30:07.038254 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:07.038183    5898 scope.go:122] "RemoveContainer" containerID="8f1d69ea8b7eb3bb7bb783daddb1907eb1df860a0171c7ba28890c649d05756d"
Oct 30 05:30:07.038548 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:30:07.038525    5898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888970-hsjd5_odh-ai-gateway-infra(b49de578-c4c0-4ae7-a090-4b9d8695b62c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888970-hsjd5" podUID="b49de578-c4c0-4ae7-a090-4b9d8695b62c"
Oct 30 05:30:07.040128 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:07.040106    5898 generic.go:356] "Generic (PLEG): container finished" podID="0f696384-5b18-4ee5-8956-a4e09dfd54f2" containerID="ad5868221654878de12844f409b65f839c2e9e4d39daa39a679239386250f654" exitCode=60
Oct 30 05:30:07.040243 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:07.040227    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-45607d-29888970-tzknr" event={"ID":"0f696384-5b18-4ee5-8956-a4e09dfd54f2","Type":"ContainerDied","Data":"ad5868221654878de12844f409b65f839c2e9e4d39daa39a679239386250f654"}
Oct 30 05:30:07.040645 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:07.040624    5898 scope.go:122] "RemoveContainer" containerID="ad5868221654878de12844f409b65f839c2e9e4d39daa39a679239386250f654"
Oct 30 05:30:07.041047 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:30:07.041025    5898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-45607d-29888970-tzknr_odh-ai-gateway-infra(0f696384-5b18-4ee5-8956-a4e09dfd54f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-45607d-29888970-tzknr" podUID="0f696384-5b18-4ee5-8956-a4e09dfd54f2"
Oct 30 05:30:07.043692 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:07.043665    5898 generic.go:356] "Generic (PLEG): container finished" podID="c687182c-2e85-4e99-8c2f-e8c972e37bcd" containerID="2ac71fdcceff1741a972fa94a0bf4950bb895e0129270d54d515072964077648" exitCode=60
Oct 30 05:30:07.043807 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:07.043785    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ec92c7-29888970-g29zl" event={"ID":"c687182c-2e85-4e99-8c2f-e8c972e37bcd","Type":"ContainerDied","Data":"2ac71fdcceff1741a972fa94a0bf4950bb895e0129270d54d515072964077648"}
Oct 30 05:30:07.044004 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:07.043987    5898 scope.go:122] "RemoveContainer" containerID="2ac71fdcceff1741a972fa94a0bf4950bb895e0129270d54d515072964077648"
Oct 30 05:30:07.044278 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:30:07.044258    5898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-ec92c7-29888970-g29zl_odh-ai-gateway-infra(c687182c-2e85-4e99-8c2f-e8c972e37bcd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ec92c7-29888970-g29zl" podUID="c687182c-2e85-4e99-8c2f-e8c972e37bcd"
Oct 30 05:30:07.045932 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:07.045889    5898 generic.go:356] "Generic (PLEG): container finished" podID="c2b36ef9-dd65-43f2-88fe-f360f18ba0fc" containerID="aae13aea4ae59b1f5b10be075615b066647696708c6b4e88801fc4efd58b000e" exitCode=60
Oct 30 05:30:07.046031 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:07.045950    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888970-22ksr" event={"ID":"c2b36ef9-dd65-43f2-88fe-f360f18ba0fc","Type":"ContainerDied","Data":"aae13aea4ae59b1f5b10be075615b066647696708c6b4e88801fc4efd58b000e"}
Oct 30 05:30:07.048074 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:07.048055    5898 generic.go:356] "Generic (PLEG): container finished" podID="2d8a177e-9a55-45dc-8b18-548547e72b6b" containerID="601bf217659d0ddee8825450f2851d9dd43ad4d7defe02603c0112348ccc8510" exitCode=60
Oct 30 05:30:07.048169 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:07.048120    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888970-k4phq" event={"ID":"2d8a177e-9a55-45dc-8b18-548547e72b6b","Type":"ContainerDied","Data":"601bf217659d0ddee8825450f2851d9dd43ad4d7defe02603c0112348ccc8510"}
Oct 30 05:30:07.048379 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:07.048364    5898 scope.go:122] "RemoveContainer" containerID="601bf217659d0ddee8825450f2851d9dd43ad4d7defe02603c0112348ccc8510"
Oct 30 05:30:07.048687 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:30:07.048662    5898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-83e83e-29888970-k4phq_odh-ai-gateway-infra(2d8a177e-9a55-45dc-8b18-548547e72b6b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888970-k4phq" podUID="2d8a177e-9a55-45dc-8b18-548547e72b6b"
Oct 30 05:30:07.050771 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:07.050749    5898 generic.go:356] "Generic (PLEG): container finished" podID="fbcc00ce-d37e-49fc-8755-7100bdfdd954" containerID="2522a38c4e5cd85df5de546edc8a2ffb7deaa56625572636a2aa48cd5e0e986f" exitCode=60
Oct 30 05:30:07.050869 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:07.050853    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2f6fed-29888970-rvzpv" event={"ID":"fbcc00ce-d37e-49fc-8755-7100bdfdd954","Type":"ContainerDied","Data":"2522a38c4e5cd85df5de546edc8a2ffb7deaa56625572636a2aa48cd5e0e986f"}
Oct 30 05:30:07.051111 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:07.051060    5898 scope.go:122] "RemoveContainer" containerID="2522a38c4e5cd85df5de546edc8a2ffb7deaa56625572636a2aa48cd5e0e986f"
Oct 30 05:30:07.051328 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:30:07.051307    5898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-2f6fed-29888970-rvzpv_odh-ai-gateway-infra(fbcc00ce-d37e-49fc-8755-7100bdfdd954)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2f6fed-29888970-rvzpv" podUID="fbcc00ce-d37e-49fc-8755-7100bdfdd954"
Oct 30 05:30:07.051450 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:07.051434    5898 scope.go:122] "RemoveContainer" containerID="aae13aea4ae59b1f5b10be075615b066647696708c6b4e88801fc4efd58b000e"
Oct 30 05:30:07.051780 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:30:07.051754    5898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-889ca9-29888970-22ksr_odh-ai-gateway-infra(c2b36ef9-dd65-43f2-88fe-f360f18ba0fc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888970-22ksr" podUID="c2b36ef9-dd65-43f2-88fe-f360f18ba0fc"
Oct 30 05:30:07.053609 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:07.053592    5898 scope.go:122] "RemoveContainer" containerID="6348f75d59a8c8b5df4baacb9fcc84d4441d8c1a92df9ea686e1d7112c2c9e15"
Oct 30 05:30:07.054189 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:07.054171    5898 generic.go:356] "Generic (PLEG): container finished" podID="08f30f4b-4298-494f-9fe3-de259a1991a0" containerID="5d356f0927897b56a3d3bb8d95a657b95ca3e3c673936fccae40deb8c022e57b" exitCode=60
Oct 30 05:30:07.054406 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:07.054385    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888970-qhwpt" event={"ID":"08f30f4b-4298-494f-9fe3-de259a1991a0","Type":"ContainerDied","Data":"5d356f0927897b56a3d3bb8d95a657b95ca3e3c673936fccae40deb8c022e57b"}
Oct 30 05:30:07.054488 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:07.054461    5898 scope.go:122] "RemoveContainer" containerID="5d356f0927897b56a3d3bb8d95a657b95ca3e3c673936fccae40deb8c022e57b"
Oct 30 05:30:07.054698 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:30:07.054673    5898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888970-qhwpt_odh-ai-gateway-infra(08f30f4b-4298-494f-9fe3-de259a1991a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888970-qhwpt" podUID="08f30f4b-4298-494f-9fe3-de259a1991a0"
Oct 30 05:30:07.059095 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:07.059077    5898 generic.go:356] "Generic (PLEG): container finished" podID="3ea1e4a0-ae9b-4a99-bf48-8f5a96e33e3b" containerID="924898dc6a58c15b456b721a72ae03bdddce9123e2c8874c10f8215b5b508a11" exitCode=60
Oct 30 05:30:07.059178 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:07.059127    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888970-5ssdn" event={"ID":"3ea1e4a0-ae9b-4a99-bf48-8f5a96e33e3b","Type":"ContainerDied","Data":"924898dc6a58c15b456b721a72ae03bdddce9123e2c8874c10f8215b5b508a11"}
Oct 30 05:30:07.059396 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:07.059378    5898 scope.go:122] "RemoveContainer" containerID="924898dc6a58c15b456b721a72ae03bdddce9123e2c8874c10f8215b5b508a11"
Oct 30 05:30:07.059634 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:30:07.059614    5898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b21507-29888970-5ssdn_odh-ai-gateway-infra(3ea1e4a0-ae9b-4a99-bf48-8f5a96e33e3b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888970-5ssdn" podUID="3ea1e4a0-ae9b-4a99-bf48-8f5a96e33e3b"
Oct 30 05:30:07.061550 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:07.061531    5898 generic.go:356] "Generic (PLEG): container finished" podID="f05b65d9-9992-445b-af08-28759a4ef2e2" containerID="e32f92fea09a45031c6a231231adc04ac51a8c240003107073418609dbe35290" exitCode=60
Oct 30 05:30:07.062421 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:07.062402    5898 scope.go:122] "RemoveContainer" containerID="e32f92fea09a45031c6a231231adc04ac51a8c240003107073418609dbe35290"
Oct 30 05:30:07.062624 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:30:07.062605    5898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888970-mf2ct_odh-ai-gateway-infra(f05b65d9-9992-445b-af08-28759a4ef2e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888970-mf2ct" podUID="f05b65d9-9992-445b-af08-28759a4ef2e2"
Oct 30 05:30:07.062710 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:07.062639    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888970-mf2ct" event={"ID":"f05b65d9-9992-445b-af08-28759a4ef2e2","Type":"ContainerDied","Data":"e32f92fea09a45031c6a231231adc04ac51a8c240003107073418609dbe35290"}
Oct 30 05:30:07.068916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:07.068882    5898 scope.go:122] "RemoveContainer" containerID="b1cfc72dbae0493f29503200dbeb7653e1f7ee7fe2451bed19936686cc951b6d"
Oct 30 05:30:07.148742 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:07.148723    5898 scope.go:122] "RemoveContainer" containerID="71a9307145839ab190ee23a55cf67b47c11fa4270cae5de6493a7c339e132c8a"
Oct 30 05:30:07.164506 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:07.164487    5898 scope.go:122] "RemoveContainer" containerID="0ad0c64aa06605ff199f42b015517be7d1d790c694e7765c16ac14e159572dda"
Oct 30 05:30:07.234452 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:07.234431    5898 scope.go:122] "RemoveContainer" containerID="01c2b90dc75cff15742145c3eb2a51064b2bcac385cdc3510c642e8f2f740469"
Oct 30 05:30:07.257319 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:07.257296    5898 scope.go:122] "RemoveContainer" containerID="de96e3d9a4ad447f7985abf8ca2e7a39b54b15f2c41225618d9457ee9d8b8aa1"
Oct 30 05:30:07.272539 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:07.272515    5898 scope.go:122] "RemoveContainer" containerID="4340094a7a27d241e8d6d101d6ac800d72c2e9892ac31e6c33d3cda2c27e10c3"
Oct 30 05:30:07.340542 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:07.340521    5898 scope.go:122] "RemoveContainer" containerID="896764cc4d48cb7d7cc99c17b1cadf04bf9c635fa355a3aa1e3c73376fb9cb8e"
Oct 30 05:30:07.355813 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:07.355793    5898 scope.go:122] "RemoveContainer" containerID="92671f7dd1a13c9b8f9ca9d032670318ca188f9488f1db79e35fb6bc9293df7d"
Oct 30 05:30:07.455351 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:07.455311    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:07.505476 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:07.505436    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Oct 30 05:30:07.505758 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:07.505726    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt" podUID="3ef747b3-e680-47d5-8849-e26bd0946d15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Oct 30 05:30:07.506519 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:07.506492    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Oct 30 05:30:07.506691 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:07.506671    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9" podUID="ee1ddb31-cd93-4f89-9968-e94fd4484b6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Oct 30 05:30:08.071641 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:08.071609    5898 scope.go:122] "RemoveContainer" containerID="aae13aea4ae59b1f5b10be075615b066647696708c6b4e88801fc4efd58b000e"
Oct 30 05:30:08.072163 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:30:08.072126    5898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-889ca9-29888970-22ksr_odh-ai-gateway-infra(c2b36ef9-dd65-43f2-88fe-f360f18ba0fc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888970-22ksr" podUID="c2b36ef9-dd65-43f2-88fe-f360f18ba0fc"
Oct 30 05:30:08.095924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:08.092051    5898 scope.go:122] "RemoveContainer" containerID="601bf217659d0ddee8825450f2851d9dd43ad4d7defe02603c0112348ccc8510"
Oct 30 05:30:08.095924 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:30:08.092558    5898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-83e83e-29888970-k4phq_odh-ai-gateway-infra(2d8a177e-9a55-45dc-8b18-548547e72b6b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888970-k4phq" podUID="2d8a177e-9a55-45dc-8b18-548547e72b6b"
Oct 30 05:30:08.119924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:08.100430    5898 scope.go:122] "RemoveContainer" containerID="2522a38c4e5cd85df5de546edc8a2ffb7deaa56625572636a2aa48cd5e0e986f"
Oct 30 05:30:08.119924 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:30:08.100856    5898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-2f6fed-29888970-rvzpv_odh-ai-gateway-infra(fbcc00ce-d37e-49fc-8755-7100bdfdd954)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2f6fed-29888970-rvzpv" podUID="fbcc00ce-d37e-49fc-8755-7100bdfdd954"
Oct 30 05:30:08.131805 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:08.131764    5898 scope.go:122] "RemoveContainer" containerID="5d356f0927897b56a3d3bb8d95a657b95ca3e3c673936fccae40deb8c022e57b"
Oct 30 05:30:08.132265 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:30:08.132232    5898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888970-qhwpt_odh-ai-gateway-infra(08f30f4b-4298-494f-9fe3-de259a1991a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888970-qhwpt" podUID="08f30f4b-4298-494f-9fe3-de259a1991a0"
Oct 30 05:30:08.139044 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:08.138594    5898 scope.go:122] "RemoveContainer" containerID="924898dc6a58c15b456b721a72ae03bdddce9123e2c8874c10f8215b5b508a11"
Oct 30 05:30:08.139044 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:30:08.138865    5898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b21507-29888970-5ssdn_odh-ai-gateway-infra(3ea1e4a0-ae9b-4a99-bf48-8f5a96e33e3b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888970-5ssdn" podUID="3ea1e4a0-ae9b-4a99-bf48-8f5a96e33e3b"
Oct 30 05:30:08.140749 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:08.140728    5898 scope.go:122] "RemoveContainer" containerID="e32f92fea09a45031c6a231231adc04ac51a8c240003107073418609dbe35290"
Oct 30 05:30:08.141082 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:30:08.141032    5898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888970-mf2ct_odh-ai-gateway-infra(f05b65d9-9992-445b-af08-28759a4ef2e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888970-mf2ct" podUID="f05b65d9-9992-445b-af08-28759a4ef2e2"
Oct 30 05:30:08.142911 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:08.142882    5898 scope.go:122] "RemoveContainer" containerID="7f6ad6199aeae680ad3e8880e44aee1ad1292a5d966b31111c773c20ca9ba026"
Oct 30 05:30:08.143138 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:30:08.143118    5898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888970-k9zdq_odh-ai-gateway-infra(7dd199d3-31ed-41c5-8570-f6a1bc1cbdc9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888970-k9zdq" podUID="7dd199d3-31ed-41c5-8570-f6a1bc1cbdc9"
Oct 30 05:30:08.145970 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:08.145951    5898 scope.go:122] "RemoveContainer" containerID="8f1d69ea8b7eb3bb7bb783daddb1907eb1df860a0171c7ba28890c649d05756d"
Oct 30 05:30:08.146260 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:30:08.146238    5898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888970-hsjd5_odh-ai-gateway-infra(b49de578-c4c0-4ae7-a090-4b9d8695b62c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888970-hsjd5" podUID="b49de578-c4c0-4ae7-a090-4b9d8695b62c"
Oct 30 05:30:08.147770 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:08.147750    5898 scope.go:122] "RemoveContainer" containerID="ad5868221654878de12844f409b65f839c2e9e4d39daa39a679239386250f654"
Oct 30 05:30:08.148027 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:30:08.148003    5898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-45607d-29888970-tzknr_odh-ai-gateway-infra(0f696384-5b18-4ee5-8956-a4e09dfd54f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-45607d-29888970-tzknr" podUID="0f696384-5b18-4ee5-8956-a4e09dfd54f2"
Oct 30 05:30:08.149323 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:08.149304    5898 scope.go:122] "RemoveContainer" containerID="2ac71fdcceff1741a972fa94a0bf4950bb895e0129270d54d515072964077648"
Oct 30 05:30:08.149592 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:30:08.149567    5898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-ec92c7-29888970-g29zl_odh-ai-gateway-infra(c687182c-2e85-4e99-8c2f-e8c972e37bcd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ec92c7-29888970-g29zl" podUID="c687182c-2e85-4e99-8c2f-e8c972e37bcd"
Oct 30 05:30:08.508001 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:08.507522    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Oct 30 05:30:08.508001 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:08.507593    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt" podUID="3ef747b3-e680-47d5-8849-e26bd0946d15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Oct 30 05:30:08.508940 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:08.508895    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Oct 30 05:30:08.509050 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:08.508971    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9" podUID="ee1ddb31-cd93-4f89-9968-e94fd4484b6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Oct 30 05:30:09.109954 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:09.109237    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-93bad9/rate-test-model-93bad9-kserve-6878b4bf49-hcjsm" podUID="66cbc6ec-813b-4904-b65e-8893ae4f02ce" containerName="main" containerID="cri-o://3b76e11779a00c3019d5462c652552f2c1ef734c7f04cda9587a07f1bf782835" gracePeriod=45
Oct 30 05:30:09.533996 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:09.530443    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Oct 30 05:30:09.533996 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:09.530554    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9" podUID="ee1ddb31-cd93-4f89-9968-e94fd4484b6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Oct 30 05:30:09.541242 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:09.534566    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:09.541242 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:09.537958    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Oct 30 05:30:09.541242 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:09.538005    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt" podUID="3ef747b3-e680-47d5-8849-e26bd0946d15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Oct 30 05:30:09.737314 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:09.737286    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-93bad9/rate-test-model-93bad9-kserve-6878b4bf49-hcjsm"
Oct 30 05:30:09.841059 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:09.841022    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66cbc6ec-813b-4904-b65e-8893ae4f02ce-kube-api-access-4g8xj\" (UniqueName: \"kubernetes.io/projected/66cbc6ec-813b-4904-b65e-8893ae4f02ce-kube-api-access-4g8xj\") pod \"66cbc6ec-813b-4904-b65e-8893ae4f02ce\" (UID: \"66cbc6ec-813b-4904-b65e-8893ae4f02ce\") "
Oct 30 05:30:09.841239 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:09.841104    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66cbc6ec-813b-4904-b65e-8893ae4f02ce-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66cbc6ec-813b-4904-b65e-8893ae4f02ce-tmp-dir\") pod \"66cbc6ec-813b-4904-b65e-8893ae4f02ce\" (UID: \"66cbc6ec-813b-4904-b65e-8893ae4f02ce\") "
Oct 30 05:30:09.841239 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:09.841147    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66cbc6ec-813b-4904-b65e-8893ae4f02ce-tls-certs\" (UniqueName: \"kubernetes.io/secret/66cbc6ec-813b-4904-b65e-8893ae4f02ce-tls-certs\") pod \"66cbc6ec-813b-4904-b65e-8893ae4f02ce\" (UID: \"66cbc6ec-813b-4904-b65e-8893ae4f02ce\") "
Oct 30 05:30:09.841378 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:09.841358    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66cbc6ec-813b-4904-b65e-8893ae4f02ce-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66cbc6ec-813b-4904-b65e-8893ae4f02ce-model-cache\") pod \"66cbc6ec-813b-4904-b65e-8893ae4f02ce\" (UID: \"66cbc6ec-813b-4904-b65e-8893ae4f02ce\") "
Oct 30 05:30:09.841688 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:09.841652    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66cbc6ec-813b-4904-b65e-8893ae4f02ce-dshm\" (UniqueName: \"kubernetes.io/empty-dir/66cbc6ec-813b-4904-b65e-8893ae4f02ce-dshm\") pod \"66cbc6ec-813b-4904-b65e-8893ae4f02ce\" (UID: \"66cbc6ec-813b-4904-b65e-8893ae4f02ce\") "
Oct 30 05:30:09.841825 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:09.841808    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66cbc6ec-813b-4904-b65e-8893ae4f02ce-home\" (UniqueName: \"kubernetes.io/empty-dir/66cbc6ec-813b-4904-b65e-8893ae4f02ce-home\") pod \"66cbc6ec-813b-4904-b65e-8893ae4f02ce\" (UID: \"66cbc6ec-813b-4904-b65e-8893ae4f02ce\") "
Oct 30 05:30:09.842653 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:09.842622    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66cbc6ec-813b-4904-b65e-8893ae4f02ce-tmp-dir" pod "66cbc6ec-813b-4904-b65e-8893ae4f02ce" (UID: "66cbc6ec-813b-4904-b65e-8893ae4f02ce"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:30:09.842745 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:09.842671    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66cbc6ec-813b-4904-b65e-8893ae4f02ce-model-cache" pod "66cbc6ec-813b-4904-b65e-8893ae4f02ce" (UID: "66cbc6ec-813b-4904-b65e-8893ae4f02ce"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:30:09.842806 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:09.842784    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66cbc6ec-813b-4904-b65e-8893ae4f02ce-home" pod "66cbc6ec-813b-4904-b65e-8893ae4f02ce" (UID: "66cbc6ec-813b-4904-b65e-8893ae4f02ce"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:30:09.844879 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:09.844857    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66cbc6ec-813b-4904-b65e-8893ae4f02ce-tls-certs" pod "66cbc6ec-813b-4904-b65e-8893ae4f02ce" (UID: "66cbc6ec-813b-4904-b65e-8893ae4f02ce"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:30:09.846620 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:09.846573    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66cbc6ec-813b-4904-b65e-8893ae4f02ce-dshm" pod "66cbc6ec-813b-4904-b65e-8893ae4f02ce" (UID: "66cbc6ec-813b-4904-b65e-8893ae4f02ce"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:30:09.847337 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:09.847200    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66cbc6ec-813b-4904-b65e-8893ae4f02ce-kube-api-access-4g8xj" pod "66cbc6ec-813b-4904-b65e-8893ae4f02ce" (UID: "66cbc6ec-813b-4904-b65e-8893ae4f02ce"). InnerVolumeSpecName "kube-api-access-4g8xj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:30:09.944291 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:09.944205    5898 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66cbc6ec-813b-4904-b65e-8893ae4f02ce-dshm\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:09.944291 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:09.944232    5898 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66cbc6ec-813b-4904-b65e-8893ae4f02ce-home\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:09.944291 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:09.944242    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g8xj\" (UniqueName: \"kubernetes.io/projected/66cbc6ec-813b-4904-b65e-8893ae4f02ce-kube-api-access-4g8xj\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:09.944291 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:09.944251    5898 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66cbc6ec-813b-4904-b65e-8893ae4f02ce-tmp-dir\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:09.944291 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:09.944259    5898 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66cbc6ec-813b-4904-b65e-8893ae4f02ce-tls-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:09.944291 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:09.944267    5898 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66cbc6ec-813b-4904-b65e-8893ae4f02ce-model-cache\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:10.168433 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:10.168395    5898 generic.go:356] "Generic (PLEG): container finished" podID="66cbc6ec-813b-4904-b65e-8893ae4f02ce" containerID="3b76e11779a00c3019d5462c652552f2c1ef734c7f04cda9587a07f1bf782835" exitCode=0
Oct 30 05:30:10.168847 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:10.168639    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-93bad9/rate-test-model-93bad9-kserve-6878b4bf49-hcjsm" event={"ID":"66cbc6ec-813b-4904-b65e-8893ae4f02ce","Type":"ContainerDied","Data":"3b76e11779a00c3019d5462c652552f2c1ef734c7f04cda9587a07f1bf782835"}
Oct 30 05:30:10.168847 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:10.168674    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-93bad9/rate-test-model-93bad9-kserve-6878b4bf49-hcjsm" event={"ID":"66cbc6ec-813b-4904-b65e-8893ae4f02ce","Type":"ContainerDied","Data":"322f8739327de730b4121a03d172438a0e2116286c7cb0a598402e41e5fcba59"}
Oct 30 05:30:10.168847 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:10.168690    5898 scope.go:122] "RemoveContainer" containerID="3b76e11779a00c3019d5462c652552f2c1ef734c7f04cda9587a07f1bf782835"
Oct 30 05:30:10.168847 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:10.168789    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-93bad9/rate-test-model-93bad9-kserve-6878b4bf49-hcjsm"
Oct 30 05:30:10.235023 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:10.234396    5898 scope.go:122] "RemoveContainer" containerID="3b76e11779a00c3019d5462c652552f2c1ef734c7f04cda9587a07f1bf782835"
Oct 30 05:30:10.235023 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:30:10.234770    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b76e11779a00c3019d5462c652552f2c1ef734c7f04cda9587a07f1bf782835\": container with ID starting with 3b76e11779a00c3019d5462c652552f2c1ef734c7f04cda9587a07f1bf782835 not found: ID does not exist" containerID="3b76e11779a00c3019d5462c652552f2c1ef734c7f04cda9587a07f1bf782835"
Oct 30 05:30:10.235023 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:10.234804    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b76e11779a00c3019d5462c652552f2c1ef734c7f04cda9587a07f1bf782835"} err="failed to get container status \"3b76e11779a00c3019d5462c652552f2c1ef734c7f04cda9587a07f1bf782835\": rpc error: code = NotFound desc = could not find container \"3b76e11779a00c3019d5462c652552f2c1ef734c7f04cda9587a07f1bf782835\": container with ID starting with 3b76e11779a00c3019d5462c652552f2c1ef734c7f04cda9587a07f1bf782835 not found: ID does not exist"
Oct 30 05:30:10.266768 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:10.266736    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-93bad9/rate-test-model-93bad9-kserve-6878b4bf49-hcjsm"]
Oct 30 05:30:10.297979 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:10.297931    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-93bad9/rate-test-model-93bad9-kserve-6878b4bf49-hcjsm"]
Oct 30 05:30:10.499992 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:10.499296    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Oct 30 05:30:10.499992 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:10.499362    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt" podUID="3ef747b3-e680-47d5-8849-e26bd0946d15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Oct 30 05:30:10.513931 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:10.513240    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Oct 30 05:30:10.513931 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:10.513299    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9" podUID="ee1ddb31-cd93-4f89-9968-e94fd4484b6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Oct 30 05:30:10.940948 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:10.940448    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66cbc6ec-813b-4904-b65e-8893ae4f02ce" path="/var/lib/kubelet/pods/66cbc6ec-813b-4904-b65e-8893ae4f02ce/volumes"
Oct 30 05:30:11.393347 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:11.393307    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:11.539443 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:11.539404    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9"
Oct 30 05:30:11.540301 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:11.540274    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9"
Oct 30 05:30:11.541306 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:11.541286    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9"
Oct 30 05:30:11.583403 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:11.583367    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt"
Oct 30 05:30:11.584124 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:11.584083    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt"
Oct 30 05:30:11.585372 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:11.585344    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt"
Oct 30 05:30:12.868252 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:12.868217    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:13.074935 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:13.065702    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:14.176795 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:14.176765    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:16.123732 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:16.123699    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:16.883364 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:16.883328    5898 scope.go:122] "RemoveContainer" containerID="924898dc6a58c15b456b721a72ae03bdddce9123e2c8874c10f8215b5b508a11"
Oct 30 05:30:16.883539 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:16.883400    5898 scope.go:122] "RemoveContainer" containerID="2ac71fdcceff1741a972fa94a0bf4950bb895e0129270d54d515072964077648"
Oct 30 05:30:16.883539 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:16.883528    5898 scope.go:122] "RemoveContainer" containerID="ad5868221654878de12844f409b65f839c2e9e4d39daa39a679239386250f654"
Oct 30 05:30:17.602389 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:17.602358    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ec92c7-29888970-g29zl" event={"ID":"c687182c-2e85-4e99-8c2f-e8c972e37bcd","Type":"ContainerStarted","Data":"a66b5d82b1c8ca3abe3edc7155c05432058a2cd702d72f042f4b3be230da908f"}
Oct 30 05:30:17.604436 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:17.604409    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888970-5ssdn" event={"ID":"3ea1e4a0-ae9b-4a99-bf48-8f5a96e33e3b","Type":"ContainerStarted","Data":"73546e9e5e8a3d4d37f794074c2596f37bca4f65d558aee8dba69c9acc95834c"}
Oct 30 05:30:17.606483 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:17.606456    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-45607d-29888970-tzknr" event={"ID":"0f696384-5b18-4ee5-8956-a4e09dfd54f2","Type":"ContainerStarted","Data":"139ee009e5cd8dd3df6897ff81ae935ba7683750c9da1c89e8025f0b614f96a9"}
Oct 30 05:30:17.683415 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:17.683362    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888970-5ssdn" podStartSLOduration=17.68334885 podStartE2EDuration="17.68334885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:17.678376907 +0000 UTC m=+2746.902196814" watchObservedRunningTime="2026-10-30 05:30:17.68334885 +0000 UTC m=+2746.907168750"
Oct 30 05:30:17.883413 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:17.883391    5898 scope.go:122] "RemoveContainer" containerID="601bf217659d0ddee8825450f2851d9dd43ad4d7defe02603c0112348ccc8510"
Oct 30 05:30:17.900861 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:17.895001    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp"]
Oct 30 05:30:17.900861 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:17.896165    5898 cpu_manager.go:414] "containerMap: removing container" podUID="66cbc6ec-813b-4904-b65e-8893ae4f02ce" containerName="main"
Oct 30 05:30:17.900861 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:17.896186    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66cbc6ec-813b-4904-b65e-8893ae4f02ce" containerName="main"
Oct 30 05:30:17.900861 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:17.896371    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="66cbc6ec-813b-4904-b65e-8893ae4f02ce" containerName="main"
Oct 30 05:30:17.901287 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:17.901166    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp"
Oct 30 05:30:17.921114 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:17.921080    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-2d21ab-openshift-default-dockercfg-85kql\""
Oct 30 05:30:17.927803 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:17.927779    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp"]
Oct 30 05:30:17.988809 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:17.988779    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80d5efb6-b3de-4f57-9249-614e1b13a04d-istio-data\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp\" (UID: \"80d5efb6-b3de-4f57-9249-614e1b13a04d\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp"
Oct 30 05:30:17.988991 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:17.988932    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80d5efb6-b3de-4f57-9249-614e1b13a04d-istio-podinfo\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp\" (UID: \"80d5efb6-b3de-4f57-9249-614e1b13a04d\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp"
Oct 30 05:30:17.989070 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:17.988992    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d5efb6-b3de-4f57-9249-614e1b13a04d-workload-socket\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp\" (UID: \"80d5efb6-b3de-4f57-9249-614e1b13a04d\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp"
Oct 30 05:30:17.989070 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:17.989049    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80d5efb6-b3de-4f57-9249-614e1b13a04d-istiod-ca-cert\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp\" (UID: \"80d5efb6-b3de-4f57-9249-614e1b13a04d\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp"
Oct 30 05:30:17.989196 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:17.989180    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80d5efb6-b3de-4f57-9249-614e1b13a04d-workload-certs\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp\" (UID: \"80d5efb6-b3de-4f57-9249-614e1b13a04d\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp"
Oct 30 05:30:17.989277 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:17.989262    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80d5efb6-b3de-4f57-9249-614e1b13a04d-istio-token\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp\" (UID: \"80d5efb6-b3de-4f57-9249-614e1b13a04d\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp"
Oct 30 05:30:17.989345 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:17.989328    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvffk\" (UniqueName: \"kubernetes.io/projected/80d5efb6-b3de-4f57-9249-614e1b13a04d-kube-api-access-hvffk\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp\" (UID: \"80d5efb6-b3de-4f57-9249-614e1b13a04d\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp"
Oct 30 05:30:17.989471 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:17.989452    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80d5efb6-b3de-4f57-9249-614e1b13a04d-istio-envoy\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp\" (UID: \"80d5efb6-b3de-4f57-9249-614e1b13a04d\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp"
Oct 30 05:30:17.989637 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:17.989617    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d5efb6-b3de-4f57-9249-614e1b13a04d-credential-socket\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp\" (UID: \"80d5efb6-b3de-4f57-9249-614e1b13a04d\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp"
Oct 30 05:30:18.090966 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:18.090937    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d5efb6-b3de-4f57-9249-614e1b13a04d-credential-socket\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp\" (UID: \"80d5efb6-b3de-4f57-9249-614e1b13a04d\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp"
Oct 30 05:30:18.091115 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:18.091060    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80d5efb6-b3de-4f57-9249-614e1b13a04d-istio-data\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp\" (UID: \"80d5efb6-b3de-4f57-9249-614e1b13a04d\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp"
Oct 30 05:30:18.091115 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:18.091108    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80d5efb6-b3de-4f57-9249-614e1b13a04d-istio-podinfo\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp\" (UID: \"80d5efb6-b3de-4f57-9249-614e1b13a04d\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp"
Oct 30 05:30:18.091196 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:18.091146    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d5efb6-b3de-4f57-9249-614e1b13a04d-workload-socket\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp\" (UID: \"80d5efb6-b3de-4f57-9249-614e1b13a04d\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp"
Oct 30 05:30:18.091196 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:18.091162    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80d5efb6-b3de-4f57-9249-614e1b13a04d-istiod-ca-cert\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp\" (UID: \"80d5efb6-b3de-4f57-9249-614e1b13a04d\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp"
Oct 30 05:30:18.091408 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:18.091388    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80d5efb6-b3de-4f57-9249-614e1b13a04d-workload-certs\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp\" (UID: \"80d5efb6-b3de-4f57-9249-614e1b13a04d\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp"
Oct 30 05:30:18.091479 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:18.091455    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d5efb6-b3de-4f57-9249-614e1b13a04d-credential-socket\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp\" (UID: \"80d5efb6-b3de-4f57-9249-614e1b13a04d\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp"
Oct 30 05:30:18.091568 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:18.091484    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80d5efb6-b3de-4f57-9249-614e1b13a04d-istio-data\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp\" (UID: \"80d5efb6-b3de-4f57-9249-614e1b13a04d\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp"
Oct 30 05:30:18.091568 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:18.091461    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80d5efb6-b3de-4f57-9249-614e1b13a04d-istio-token\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp\" (UID: \"80d5efb6-b3de-4f57-9249-614e1b13a04d\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp"
Oct 30 05:30:18.091641 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:18.091599    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvffk\" (UniqueName: \"kubernetes.io/projected/80d5efb6-b3de-4f57-9249-614e1b13a04d-kube-api-access-hvffk\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp\" (UID: \"80d5efb6-b3de-4f57-9249-614e1b13a04d\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp"
Oct 30 05:30:18.091678 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:18.091643    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80d5efb6-b3de-4f57-9249-614e1b13a04d-istio-envoy\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp\" (UID: \"80d5efb6-b3de-4f57-9249-614e1b13a04d\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp"
Oct 30 05:30:18.091870 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:18.091847    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d5efb6-b3de-4f57-9249-614e1b13a04d-workload-socket\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp\" (UID: \"80d5efb6-b3de-4f57-9249-614e1b13a04d\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp"
Oct 30 05:30:18.092088 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:18.092063    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80d5efb6-b3de-4f57-9249-614e1b13a04d-workload-certs\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp\" (UID: \"80d5efb6-b3de-4f57-9249-614e1b13a04d\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp"
Oct 30 05:30:18.092228 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:18.092207    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80d5efb6-b3de-4f57-9249-614e1b13a04d-istiod-ca-cert\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp\" (UID: \"80d5efb6-b3de-4f57-9249-614e1b13a04d\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp"
Oct 30 05:30:18.093837 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:18.093816    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80d5efb6-b3de-4f57-9249-614e1b13a04d-istio-envoy\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp\" (UID: \"80d5efb6-b3de-4f57-9249-614e1b13a04d\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp"
Oct 30 05:30:18.093932 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:18.093816    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80d5efb6-b3de-4f57-9249-614e1b13a04d-istio-podinfo\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp\" (UID: \"80d5efb6-b3de-4f57-9249-614e1b13a04d\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp"
Oct 30 05:30:18.110679 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:18.110654    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvffk\" (UniqueName: \"kubernetes.io/projected/80d5efb6-b3de-4f57-9249-614e1b13a04d-kube-api-access-hvffk\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp\" (UID: \"80d5efb6-b3de-4f57-9249-614e1b13a04d\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp"
Oct 30 05:30:18.110781 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:18.110675    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80d5efb6-b3de-4f57-9249-614e1b13a04d-istio-token\") pod \"e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp\" (UID: \"80d5efb6-b3de-4f57-9249-614e1b13a04d\") " pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp"
Oct 30 05:30:18.144395 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:18.144372    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp"
Oct 30 05:30:18.480166 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:18.480142    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp"]
Oct 30 05:30:18.481018 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:30:18.480993    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80d5efb6_b3de_4f57_9249_614e1b13a04d.slice/crio-cd7ac96374efaa432e44b06869336d0ae9e0d1dc855daebbd5f6d56d4a02fe2c WatchSource:0}: Error finding container cd7ac96374efaa432e44b06869336d0ae9e0d1dc855daebbd5f6d56d4a02fe2c: Status 404 returned error can't find the container with id cd7ac96374efaa432e44b06869336d0ae9e0d1dc855daebbd5f6d56d4a02fe2c
Oct 30 05:30:18.484468 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:18.484435    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:30:18.484579 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:18.484502    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:30:18.484579 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:18.484531    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:30:18.536109 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:18.536084    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:18.668218 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:18.666820    5898 generic.go:356] "Generic (PLEG): container finished" podID="3ea1e4a0-ae9b-4a99-bf48-8f5a96e33e3b" containerID="73546e9e5e8a3d4d37f794074c2596f37bca4f65d558aee8dba69c9acc95834c" exitCode=60
Oct 30 05:30:18.668218 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:18.667035    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888970-5ssdn" event={"ID":"3ea1e4a0-ae9b-4a99-bf48-8f5a96e33e3b","Type":"ContainerDied","Data":"73546e9e5e8a3d4d37f794074c2596f37bca4f65d558aee8dba69c9acc95834c"}
Oct 30 05:30:18.668218 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:18.667095    5898 scope.go:122] "RemoveContainer" containerID="924898dc6a58c15b456b721a72ae03bdddce9123e2c8874c10f8215b5b508a11"
Oct 30 05:30:18.668218 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:18.667810    5898 scope.go:122] "RemoveContainer" containerID="73546e9e5e8a3d4d37f794074c2596f37bca4f65d558aee8dba69c9acc95834c"
Oct 30 05:30:18.668871 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:30:18.668381    5898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b21507-29888970-5ssdn_odh-ai-gateway-infra(3ea1e4a0-ae9b-4a99-bf48-8f5a96e33e3b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888970-5ssdn" podUID="3ea1e4a0-ae9b-4a99-bf48-8f5a96e33e3b"
Oct 30 05:30:18.678835 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:18.676505    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ec92c7-29888970-g29zl"]
Oct 30 05:30:18.678835 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:18.678027    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp" event={"ID":"80d5efb6-b3de-4f57-9249-614e1b13a04d","Type":"ContainerStarted","Data":"2425a88d4d75ad63aafca1dc1b93f9306645314872441fc127e1d6a6f2e88389"}
Oct 30 05:30:18.678835 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:18.678077    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp" event={"ID":"80d5efb6-b3de-4f57-9249-614e1b13a04d","Type":"ContainerStarted","Data":"cd7ac96374efaa432e44b06869336d0ae9e0d1dc855daebbd5f6d56d4a02fe2c"}
Oct 30 05:30:18.683920 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:18.682214    5898 generic.go:356] "Generic (PLEG): container finished" podID="0f696384-5b18-4ee5-8956-a4e09dfd54f2" containerID="139ee009e5cd8dd3df6897ff81ae935ba7683750c9da1c89e8025f0b614f96a9" exitCode=60
Oct 30 05:30:18.683920 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:18.682240    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-45607d-29888970-tzknr" event={"ID":"0f696384-5b18-4ee5-8956-a4e09dfd54f2","Type":"ContainerDied","Data":"139ee009e5cd8dd3df6897ff81ae935ba7683750c9da1c89e8025f0b614f96a9"}
Oct 30 05:30:18.683920 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:18.683201    5898 scope.go:122] "RemoveContainer" containerID="139ee009e5cd8dd3df6897ff81ae935ba7683750c9da1c89e8025f0b614f96a9"
Oct 30 05:30:18.685959 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:30:18.684620    5898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-45607d-29888970-tzknr_odh-ai-gateway-infra(0f696384-5b18-4ee5-8956-a4e09dfd54f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-45607d-29888970-tzknr" podUID="0f696384-5b18-4ee5-8956-a4e09dfd54f2"
Oct 30 05:30:18.687078 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:18.687051    5898 generic.go:356] "Generic (PLEG): container finished" podID="c687182c-2e85-4e99-8c2f-e8c972e37bcd" containerID="a66b5d82b1c8ca3abe3edc7155c05432058a2cd702d72f042f4b3be230da908f" exitCode=60
Oct 30 05:30:18.687232 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:18.687214    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ec92c7-29888970-g29zl" event={"ID":"c687182c-2e85-4e99-8c2f-e8c972e37bcd","Type":"ContainerDied","Data":"a66b5d82b1c8ca3abe3edc7155c05432058a2cd702d72f042f4b3be230da908f"}
Oct 30 05:30:18.690672 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:18.690646    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888970-k4phq" event={"ID":"2d8a177e-9a55-45dc-8b18-548547e72b6b","Type":"ContainerStarted","Data":"88a1f4e4347888c0955aa621894b2562d5c4004ea11fa41257d420d77c55917e"}
Oct 30 05:30:18.696006 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:18.695925    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888970-5ssdn"]
Oct 30 05:30:18.704057 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:18.704028    5898 scope.go:122] "RemoveContainer" containerID="ad5868221654878de12844f409b65f839c2e9e4d39daa39a679239386250f654"
Oct 30 05:30:18.726307 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:18.726282    5898 scope.go:122] "RemoveContainer" containerID="2ac71fdcceff1741a972fa94a0bf4950bb895e0129270d54d515072964077648"
Oct 30 05:30:18.740161 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:18.740126    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-45607d-29888970-tzknr"]
Oct 30 05:30:18.817308 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:18.816202    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp" podStartSLOduration=1.816180252 podStartE2EDuration="1.816180252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:18.761037649 +0000 UTC m=+2747.984857551" watchObservedRunningTime="2026-10-30 05:30:18.816180252 +0000 UTC m=+2748.040000153"
Oct 30 05:30:18.888368 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:18.888337    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ec92c7-29888970-g29zl"
Oct 30 05:30:18.895190 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:18.895164    5898 scope.go:122] "RemoveContainer" containerID="8f1d69ea8b7eb3bb7bb783daddb1907eb1df860a0171c7ba28890c649d05756d"
Oct 30 05:30:18.895510 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:18.895495    5898 scope.go:122] "RemoveContainer" containerID="2522a38c4e5cd85df5de546edc8a2ffb7deaa56625572636a2aa48cd5e0e986f"
Oct 30 05:30:19.016127 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:19.016093    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c687182c-2e85-4e99-8c2f-e8c972e37bcd-kube-api-access-kxfwf\" (UniqueName: \"kubernetes.io/projected/c687182c-2e85-4e99-8c2f-e8c972e37bcd-kube-api-access-kxfwf\") pod \"c687182c-2e85-4e99-8c2f-e8c972e37bcd\" (UID: \"c687182c-2e85-4e99-8c2f-e8c972e37bcd\") "
Oct 30 05:30:19.016312 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:19.016231    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c687182c-2e85-4e99-8c2f-e8c972e37bcd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c687182c-2e85-4e99-8c2f-e8c972e37bcd-maas-api-service-ca\") pod \"c687182c-2e85-4e99-8c2f-e8c972e37bcd\" (UID: \"c687182c-2e85-4e99-8c2f-e8c972e37bcd\") "
Oct 30 05:30:19.016875 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:19.016841    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c687182c-2e85-4e99-8c2f-e8c972e37bcd-maas-api-service-ca" pod "c687182c-2e85-4e99-8c2f-e8c972e37bcd" (UID: "c687182c-2e85-4e99-8c2f-e8c972e37bcd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:30:19.017197 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:19.017173    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c687182c-2e85-4e99-8c2f-e8c972e37bcd-maas-api-service-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:19.018833 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:19.018812    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c687182c-2e85-4e99-8c2f-e8c972e37bcd-kube-api-access-kxfwf" pod "c687182c-2e85-4e99-8c2f-e8c972e37bcd" (UID: "c687182c-2e85-4e99-8c2f-e8c972e37bcd"). InnerVolumeSpecName "kube-api-access-kxfwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:30:19.118396 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:19.118369    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxfwf\" (UniqueName: \"kubernetes.io/projected/c687182c-2e85-4e99-8c2f-e8c972e37bcd-kube-api-access-kxfwf\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:19.145558 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:19.145528    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp"
Oct 30 05:30:19.146769 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:19.146746    5898 patch_prober.go:29] interesting pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Oct 30 05:30:19.146872 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:19.146790    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp" podUID="80d5efb6-b3de-4f57-9249-614e1b13a04d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Oct 30 05:30:19.714756 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:19.714710    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ec92c7-29888970-g29zl"
Oct 30 05:30:19.714756 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:19.714747    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ec92c7-29888970-g29zl" event={"ID":"c687182c-2e85-4e99-8c2f-e8c972e37bcd","Type":"ContainerDied","Data":"8cb8cc05d77f6f81b48aef868379828258f13fcc9c85d3977d3fe5e3baeec7f2"}
Oct 30 05:30:19.715368 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:19.714806    5898 scope.go:122] "RemoveContainer" containerID="a66b5d82b1c8ca3abe3edc7155c05432058a2cd702d72f042f4b3be230da908f"
Oct 30 05:30:19.717246 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:19.717222    5898 generic.go:356] "Generic (PLEG): container finished" podID="2d8a177e-9a55-45dc-8b18-548547e72b6b" containerID="88a1f4e4347888c0955aa621894b2562d5c4004ea11fa41257d420d77c55917e" exitCode=60
Oct 30 05:30:19.717364 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:19.717306    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888970-k4phq" event={"ID":"2d8a177e-9a55-45dc-8b18-548547e72b6b","Type":"ContainerDied","Data":"88a1f4e4347888c0955aa621894b2562d5c4004ea11fa41257d420d77c55917e"}
Oct 30 05:30:19.717852 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:19.717832    5898 scope.go:122] "RemoveContainer" containerID="88a1f4e4347888c0955aa621894b2562d5c4004ea11fa41257d420d77c55917e"
Oct 30 05:30:19.718332 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:30:19.718305    5898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-83e83e-29888970-k4phq_odh-ai-gateway-infra(2d8a177e-9a55-45dc-8b18-548547e72b6b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888970-k4phq" podUID="2d8a177e-9a55-45dc-8b18-548547e72b6b"
Oct 30 05:30:19.720136 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:19.720111    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2f6fed-29888970-rvzpv" event={"ID":"fbcc00ce-d37e-49fc-8755-7100bdfdd954","Type":"ContainerStarted","Data":"2ecd8a11ecfeb9b96301b05200236a834cbae6184a1c5a5d6a309a9c7b6cdc46"}
Oct 30 05:30:19.724541 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:19.724360    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888970-hsjd5" event={"ID":"b49de578-c4c0-4ae7-a090-4b9d8695b62c","Type":"ContainerStarted","Data":"655fd08b217c517364a08387897999ec6998889a152a1acd849e4c2ad9397dd9"}
Oct 30 05:30:19.727648 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:19.727630    5898 scope.go:122] "RemoveContainer" containerID="601bf217659d0ddee8825450f2851d9dd43ad4d7defe02603c0112348ccc8510"
Oct 30 05:30:19.787294 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:19.773798    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ec92c7-29888970-g29zl"]
Oct 30 05:30:19.812178 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:19.812068    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ec92c7-29888970-g29zl"]
Oct 30 05:30:19.905043 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:19.898164    5898 scope.go:122] "RemoveContainer" containerID="7f6ad6199aeae680ad3e8880e44aee1ad1292a5d966b31111c773c20ca9ba026"
Oct 30 05:30:19.911190 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:19.909424    5898 scope.go:122] "RemoveContainer" containerID="e32f92fea09a45031c6a231231adc04ac51a8c240003107073418609dbe35290"
Oct 30 05:30:19.911190 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:19.910681    5898 scope.go:122] "RemoveContainer" containerID="5d356f0927897b56a3d3bb8d95a657b95ca3e3c673936fccae40deb8c022e57b"
Oct 30 05:30:19.911977 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:19.911893    5898 scope.go:122] "RemoveContainer" containerID="aae13aea4ae59b1f5b10be075615b066647696708c6b4e88801fc4efd58b000e"
Oct 30 05:30:19.924045 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:19.923945    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888970-k4phq"]
Oct 30 05:30:20.134441 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:20.134413    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888970-5ssdn"
Oct 30 05:30:20.144681 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:20.144652    5898 patch_prober.go:29] interesting pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Oct 30 05:30:20.144789 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:20.144693    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp" podUID="80d5efb6-b3de-4f57-9249-614e1b13a04d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Oct 30 05:30:20.164518 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:20.164497    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ea1e4a0-ae9b-4a99-bf48-8f5a96e33e3b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ea1e4a0-ae9b-4a99-bf48-8f5a96e33e3b-maas-api-service-ca\") pod \"3ea1e4a0-ae9b-4a99-bf48-8f5a96e33e3b\" (UID: \"3ea1e4a0-ae9b-4a99-bf48-8f5a96e33e3b\") "
Oct 30 05:30:20.164809 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:20.164792    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ea1e4a0-ae9b-4a99-bf48-8f5a96e33e3b-kube-api-access-czz75\" (UniqueName: \"kubernetes.io/projected/3ea1e4a0-ae9b-4a99-bf48-8f5a96e33e3b-kube-api-access-czz75\") pod \"3ea1e4a0-ae9b-4a99-bf48-8f5a96e33e3b\" (UID: \"3ea1e4a0-ae9b-4a99-bf48-8f5a96e33e3b\") "
Oct 30 05:30:20.164890 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:20.164831    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ea1e4a0-ae9b-4a99-bf48-8f5a96e33e3b-maas-api-service-ca" pod "3ea1e4a0-ae9b-4a99-bf48-8f5a96e33e3b" (UID: "3ea1e4a0-ae9b-4a99-bf48-8f5a96e33e3b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:30:20.165343 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:20.165326    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ea1e4a0-ae9b-4a99-bf48-8f5a96e33e3b-maas-api-service-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:20.165698 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:20.165678    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-45607d-29888970-tzknr"
Oct 30 05:30:20.167043 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:20.167018    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ea1e4a0-ae9b-4a99-bf48-8f5a96e33e3b-kube-api-access-czz75" pod "3ea1e4a0-ae9b-4a99-bf48-8f5a96e33e3b" (UID: "3ea1e4a0-ae9b-4a99-bf48-8f5a96e33e3b"). InnerVolumeSpecName "kube-api-access-czz75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:30:20.266383 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:20.266351    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f696384-5b18-4ee5-8956-a4e09dfd54f2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f696384-5b18-4ee5-8956-a4e09dfd54f2-maas-api-service-ca\") pod \"0f696384-5b18-4ee5-8956-a4e09dfd54f2\" (UID: \"0f696384-5b18-4ee5-8956-a4e09dfd54f2\") "
Oct 30 05:30:20.266580 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:20.266496    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f696384-5b18-4ee5-8956-a4e09dfd54f2-kube-api-access-h8v6k\" (UniqueName: \"kubernetes.io/projected/0f696384-5b18-4ee5-8956-a4e09dfd54f2-kube-api-access-h8v6k\") pod \"0f696384-5b18-4ee5-8956-a4e09dfd54f2\" (UID: \"0f696384-5b18-4ee5-8956-a4e09dfd54f2\") "
Oct 30 05:30:20.267321 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:20.267293    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czz75\" (UniqueName: \"kubernetes.io/projected/3ea1e4a0-ae9b-4a99-bf48-8f5a96e33e3b-kube-api-access-czz75\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:20.267468 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:20.267449    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f696384-5b18-4ee5-8956-a4e09dfd54f2-maas-api-service-ca" pod "0f696384-5b18-4ee5-8956-a4e09dfd54f2" (UID: "0f696384-5b18-4ee5-8956-a4e09dfd54f2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:30:20.305455 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:20.305401    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f696384-5b18-4ee5-8956-a4e09dfd54f2-kube-api-access-h8v6k" pod "0f696384-5b18-4ee5-8956-a4e09dfd54f2" (UID: "0f696384-5b18-4ee5-8956-a4e09dfd54f2"). InnerVolumeSpecName "kube-api-access-h8v6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:30:20.368969 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:20.368941    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f696384-5b18-4ee5-8956-a4e09dfd54f2-maas-api-service-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:20.369081 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:20.368974    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8v6k\" (UniqueName: \"kubernetes.io/projected/0f696384-5b18-4ee5-8956-a4e09dfd54f2-kube-api-access-h8v6k\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:20.545220 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:20.545131    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:20.748187 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:20.748150    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888970-k9zdq" event={"ID":"7dd199d3-31ed-41c5-8570-f6a1bc1cbdc9","Type":"ContainerStarted","Data":"b59df11c0c971052811a152cbfaa8804388c571600a7c2f09e21497da6bb1169"}
Oct 30 05:30:20.755926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:20.752861    5898 generic.go:356] "Generic (PLEG): container finished" podID="b49de578-c4c0-4ae7-a090-4b9d8695b62c" containerID="655fd08b217c517364a08387897999ec6998889a152a1acd849e4c2ad9397dd9" exitCode=60
Oct 30 05:30:20.755926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:20.753004    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888970-hsjd5" event={"ID":"b49de578-c4c0-4ae7-a090-4b9d8695b62c","Type":"ContainerDied","Data":"655fd08b217c517364a08387897999ec6998889a152a1acd849e4c2ad9397dd9"}
Oct 30 05:30:20.755926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:20.753054    5898 scope.go:122] "RemoveContainer" containerID="8f1d69ea8b7eb3bb7bb783daddb1907eb1df860a0171c7ba28890c649d05756d"
Oct 30 05:30:20.755926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:20.753435    5898 scope.go:122] "RemoveContainer" containerID="655fd08b217c517364a08387897999ec6998889a152a1acd849e4c2ad9397dd9"
Oct 30 05:30:20.755926 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:30:20.753741    5898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888970-hsjd5_odh-ai-gateway-infra(b49de578-c4c0-4ae7-a090-4b9d8695b62c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888970-hsjd5" podUID="b49de578-c4c0-4ae7-a090-4b9d8695b62c"
Oct 30 05:30:20.755926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:20.755547    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-45607d-29888970-tzknr"
Oct 30 05:30:20.757038 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:20.756988    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-45607d-29888970-tzknr" event={"ID":"0f696384-5b18-4ee5-8956-a4e09dfd54f2","Type":"ContainerDied","Data":"f2540c055717613d8661afea8cde03407c3f0309d37964f6f22e79178e4a3d01"}
Oct 30 05:30:20.766344 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:20.765724    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888970-22ksr" event={"ID":"c2b36ef9-dd65-43f2-88fe-f360f18ba0fc","Type":"ContainerStarted","Data":"915419b423a9fe6db5162485f22b3619447f321c713ffa44e88d6b8a462d8f41"}
Oct 30 05:30:20.771591 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:20.771532    5898 scope.go:122] "RemoveContainer" containerID="139ee009e5cd8dd3df6897ff81ae935ba7683750c9da1c89e8025f0b614f96a9"
Oct 30 05:30:20.773267 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:20.773244    5898 generic.go:356] "Generic (PLEG): container finished" podID="fbcc00ce-d37e-49fc-8755-7100bdfdd954" containerID="2ecd8a11ecfeb9b96301b05200236a834cbae6184a1c5a5d6a309a9c7b6cdc46" exitCode=60
Oct 30 05:30:20.773363 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:20.773310    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2f6fed-29888970-rvzpv" event={"ID":"fbcc00ce-d37e-49fc-8755-7100bdfdd954","Type":"ContainerDied","Data":"2ecd8a11ecfeb9b96301b05200236a834cbae6184a1c5a5d6a309a9c7b6cdc46"}
Oct 30 05:30:20.773716 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:20.773693    5898 scope.go:122] "RemoveContainer" containerID="2ecd8a11ecfeb9b96301b05200236a834cbae6184a1c5a5d6a309a9c7b6cdc46"
Oct 30 05:30:20.783041 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:30:20.781129    5898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-2f6fed-29888970-rvzpv_odh-ai-gateway-infra(fbcc00ce-d37e-49fc-8755-7100bdfdd954)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2f6fed-29888970-rvzpv" podUID="fbcc00ce-d37e-49fc-8755-7100bdfdd954"
Oct 30 05:30:20.807408 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:20.807333    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888970-qhwpt" event={"ID":"08f30f4b-4298-494f-9fe3-de259a1991a0","Type":"ContainerStarted","Data":"961d2c947a825cc3c62379feca34274e5e434aa0c4fb49320349d8ee08395c72"}
Oct 30 05:30:20.810165 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:20.810128    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888970-5ssdn"
Oct 30 05:30:20.812194 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:20.812167    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888970-5ssdn" event={"ID":"3ea1e4a0-ae9b-4a99-bf48-8f5a96e33e3b","Type":"ContainerDied","Data":"4867542f1fd5f406c92eee5275e8942ba6a952cdc823150c72faa47eb9d64617"}
Oct 30 05:30:20.816482 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:20.816467    5898 scope.go:122] "RemoveContainer" containerID="2522a38c4e5cd85df5de546edc8a2ffb7deaa56625572636a2aa48cd5e0e986f"
Oct 30 05:30:20.819584 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:20.819547    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888970-mf2ct" event={"ID":"f05b65d9-9992-445b-af08-28759a4ef2e2","Type":"ContainerStarted","Data":"122582ec94f99ae39139b25506c5190bb5254d9e879d5c2677eea55cca9d523a"}
Oct 30 05:30:20.846832 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:20.846809    5898 scope.go:122] "RemoveContainer" containerID="73546e9e5e8a3d4d37f794074c2596f37bca4f65d558aee8dba69c9acc95834c"
Oct 30 05:30:20.873826 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:20.873798    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888970-hsjd5"]
Oct 30 05:30:20.959702 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:20.959663    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c687182c-2e85-4e99-8c2f-e8c972e37bcd" path="/var/lib/kubelet/pods/c687182c-2e85-4e99-8c2f-e8c972e37bcd/volumes"
Oct 30 05:30:20.971725 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:20.971668    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-45607d-29888970-tzknr"]
Oct 30 05:30:21.015427 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:21.015393    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-45607d-29888970-tzknr"]
Oct 30 05:30:21.029696 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:21.029664    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2f6fed-29888970-rvzpv"]
Oct 30 05:30:21.117124 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:21.117087    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888970-k4phq"
Oct 30 05:30:21.122041 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:21.121955    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888970-5ssdn"]
Oct 30 05:30:21.126060 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:21.125975    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b21507-29888970-5ssdn"]
Oct 30 05:30:21.145520 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:21.145485    5898 patch_prober.go:29] interesting pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Oct 30 05:30:21.145700 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:21.145543    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp" podUID="80d5efb6-b3de-4f57-9249-614e1b13a04d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Oct 30 05:30:21.220202 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:21.220171    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d8a177e-9a55-45dc-8b18-548547e72b6b-kube-api-access-qxqtq\" (UniqueName: \"kubernetes.io/projected/2d8a177e-9a55-45dc-8b18-548547e72b6b-kube-api-access-qxqtq\") pod \"2d8a177e-9a55-45dc-8b18-548547e72b6b\" (UID: \"2d8a177e-9a55-45dc-8b18-548547e72b6b\") "
Oct 30 05:30:21.220372 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:21.220325    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d8a177e-9a55-45dc-8b18-548547e72b6b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d8a177e-9a55-45dc-8b18-548547e72b6b-maas-api-service-ca\") pod \"2d8a177e-9a55-45dc-8b18-548547e72b6b\" (UID: \"2d8a177e-9a55-45dc-8b18-548547e72b6b\") "
Oct 30 05:30:21.220814 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:21.220782    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d8a177e-9a55-45dc-8b18-548547e72b6b-maas-api-service-ca" pod "2d8a177e-9a55-45dc-8b18-548547e72b6b" (UID: "2d8a177e-9a55-45dc-8b18-548547e72b6b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:30:21.221083 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:21.221055    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d8a177e-9a55-45dc-8b18-548547e72b6b-maas-api-service-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:21.222684 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:21.222655    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d8a177e-9a55-45dc-8b18-548547e72b6b-kube-api-access-qxqtq" pod "2d8a177e-9a55-45dc-8b18-548547e72b6b" (UID: "2d8a177e-9a55-45dc-8b18-548547e72b6b"). InnerVolumeSpecName "kube-api-access-qxqtq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:30:21.329445 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:21.329399    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxqtq\" (UniqueName: \"kubernetes.io/projected/2d8a177e-9a55-45dc-8b18-548547e72b6b-kube-api-access-qxqtq\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:21.811239 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:21.811151    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888970-k9zdq"]
Oct 30 05:30:21.828457 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:21.828422    5898 generic.go:356] "Generic (PLEG): container finished" podID="08f30f4b-4298-494f-9fe3-de259a1991a0" containerID="961d2c947a825cc3c62379feca34274e5e434aa0c4fb49320349d8ee08395c72" exitCode=60
Oct 30 05:30:21.828615 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:21.828498    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888970-qhwpt" event={"ID":"08f30f4b-4298-494f-9fe3-de259a1991a0","Type":"ContainerDied","Data":"961d2c947a825cc3c62379feca34274e5e434aa0c4fb49320349d8ee08395c72"}
Oct 30 05:30:21.828615 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:21.828556    5898 scope.go:122] "RemoveContainer" containerID="5d356f0927897b56a3d3bb8d95a657b95ca3e3c673936fccae40deb8c022e57b"
Oct 30 05:30:21.829011 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:21.828985    5898 scope.go:122] "RemoveContainer" containerID="961d2c947a825cc3c62379feca34274e5e434aa0c4fb49320349d8ee08395c72"
Oct 30 05:30:21.829417 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:30:21.829394    5898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888970-qhwpt_odh-ai-gateway-infra(08f30f4b-4298-494f-9fe3-de259a1991a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888970-qhwpt" podUID="08f30f4b-4298-494f-9fe3-de259a1991a0"
Oct 30 05:30:21.833842 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:21.833817    5898 generic.go:356] "Generic (PLEG): container finished" podID="f05b65d9-9992-445b-af08-28759a4ef2e2" containerID="122582ec94f99ae39139b25506c5190bb5254d9e879d5c2677eea55cca9d523a" exitCode=60
Oct 30 05:30:21.833992 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:21.833895    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888970-mf2ct" event={"ID":"f05b65d9-9992-445b-af08-28759a4ef2e2","Type":"ContainerDied","Data":"122582ec94f99ae39139b25506c5190bb5254d9e879d5c2677eea55cca9d523a"}
Oct 30 05:30:21.834409 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:21.834387    5898 scope.go:122] "RemoveContainer" containerID="122582ec94f99ae39139b25506c5190bb5254d9e879d5c2677eea55cca9d523a"
Oct 30 05:30:21.834842 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:30:21.834814    5898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888970-mf2ct_odh-ai-gateway-infra(f05b65d9-9992-445b-af08-28759a4ef2e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888970-mf2ct" podUID="f05b65d9-9992-445b-af08-28759a4ef2e2"
Oct 30 05:30:21.837211 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:21.837186    5898 generic.go:356] "Generic (PLEG): container finished" podID="7dd199d3-31ed-41c5-8570-f6a1bc1cbdc9" containerID="b59df11c0c971052811a152cbfaa8804388c571600a7c2f09e21497da6bb1169" exitCode=60
Oct 30 05:30:21.837324 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:21.837307    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888970-k9zdq" event={"ID":"7dd199d3-31ed-41c5-8570-f6a1bc1cbdc9","Type":"ContainerDied","Data":"b59df11c0c971052811a152cbfaa8804388c571600a7c2f09e21497da6bb1169"}
Oct 30 05:30:21.844510 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:21.844487    5898 generic.go:356] "Generic (PLEG): container finished" podID="c2b36ef9-dd65-43f2-88fe-f360f18ba0fc" containerID="915419b423a9fe6db5162485f22b3619447f321c713ffa44e88d6b8a462d8f41" exitCode=60
Oct 30 05:30:21.844612 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:21.844562    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888970-22ksr" event={"ID":"c2b36ef9-dd65-43f2-88fe-f360f18ba0fc","Type":"ContainerDied","Data":"915419b423a9fe6db5162485f22b3619447f321c713ffa44e88d6b8a462d8f41"}
Oct 30 05:30:21.845061 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:21.845039    5898 scope.go:122] "RemoveContainer" containerID="915419b423a9fe6db5162485f22b3619447f321c713ffa44e88d6b8a462d8f41"
Oct 30 05:30:21.845402 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:30:21.845375    5898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-889ca9-29888970-22ksr_odh-ai-gateway-infra(c2b36ef9-dd65-43f2-88fe-f360f18ba0fc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888970-22ksr" podUID="c2b36ef9-dd65-43f2-88fe-f360f18ba0fc"
Oct 30 05:30:21.846115 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:21.846093    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888970-k4phq"
Oct 30 05:30:21.846115 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:21.846103    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888970-k4phq" event={"ID":"2d8a177e-9a55-45dc-8b18-548547e72b6b","Type":"ContainerDied","Data":"5acae268574b2cde45a4d95a2f68694edb3217ecadf1077e5fdec6f2a39eb75c"}
Oct 30 05:30:21.846483 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:21.846467    5898 scope.go:122] "RemoveContainer" containerID="e32f92fea09a45031c6a231231adc04ac51a8c240003107073418609dbe35290"
Oct 30 05:30:21.869557 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:21.869524    5898 scope.go:122] "RemoveContainer" containerID="7f6ad6199aeae680ad3e8880e44aee1ad1292a5d966b31111c773c20ca9ba026"
Oct 30 05:30:21.927098 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:21.927068    5898 scope.go:122] "RemoveContainer" containerID="aae13aea4ae59b1f5b10be075615b066647696708c6b4e88801fc4efd58b000e"
Oct 30 05:30:21.970418 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:21.970054    5898 scope.go:122] "RemoveContainer" containerID="88a1f4e4347888c0955aa621894b2562d5c4004ea11fa41257d420d77c55917e"
Oct 30 05:30:22.021096 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.020934    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888970-k4phq"]
Oct 30 05:30:22.033209 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.033178    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-83e83e-29888970-k4phq"]
Oct 30 05:30:22.039311 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.039283    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888970-22ksr"]
Oct 30 05:30:22.149502 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.149107    5898 patch_prober.go:29] interesting pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Oct 30 05:30:22.149502 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.149161    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp" podUID="80d5efb6-b3de-4f57-9249-614e1b13a04d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Oct 30 05:30:22.162895 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.162805    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888970-mf2ct"]
Oct 30 05:30:22.215986 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.215919    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888970-k9zdq"
Oct 30 05:30:22.225466 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.225385    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888970-qhwpt"]
Oct 30 05:30:22.322651 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.322621    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888970-hsjd5"
Oct 30 05:30:22.328642 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.328599    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2f6fed-29888970-rvzpv"
Oct 30 05:30:22.356403 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.356374    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dd199d3-31ed-41c5-8570-f6a1bc1cbdc9-kube-api-access-g22rz\" (UniqueName: \"kubernetes.io/projected/7dd199d3-31ed-41c5-8570-f6a1bc1cbdc9-kube-api-access-g22rz\") pod \"7dd199d3-31ed-41c5-8570-f6a1bc1cbdc9\" (UID: \"7dd199d3-31ed-41c5-8570-f6a1bc1cbdc9\") "
Oct 30 05:30:22.356684 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.356670    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dd199d3-31ed-41c5-8570-f6a1bc1cbdc9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dd199d3-31ed-41c5-8570-f6a1bc1cbdc9-maas-api-service-ca\") pod \"7dd199d3-31ed-41c5-8570-f6a1bc1cbdc9\" (UID: \"7dd199d3-31ed-41c5-8570-f6a1bc1cbdc9\") "
Oct 30 05:30:22.357036 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.357016    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dd199d3-31ed-41c5-8570-f6a1bc1cbdc9-maas-api-service-ca" pod "7dd199d3-31ed-41c5-8570-f6a1bc1cbdc9" (UID: "7dd199d3-31ed-41c5-8570-f6a1bc1cbdc9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:30:22.357175 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.357161    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dd199d3-31ed-41c5-8570-f6a1bc1cbdc9-maas-api-service-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:22.358438 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.358423    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dd199d3-31ed-41c5-8570-f6a1bc1cbdc9-kube-api-access-g22rz" pod "7dd199d3-31ed-41c5-8570-f6a1bc1cbdc9" (UID: "7dd199d3-31ed-41c5-8570-f6a1bc1cbdc9"). InnerVolumeSpecName "kube-api-access-g22rz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:30:22.413361 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.413279    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b"]
Oct 30 05:30:22.414640 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.414609    5898 cpu_manager.go:414] "containerMap: removing container" podUID="fbcc00ce-d37e-49fc-8755-7100bdfdd954" containerName="cleanup"
Oct 30 05:30:22.414640 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.414640    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbcc00ce-d37e-49fc-8755-7100bdfdd954" containerName="cleanup"
Oct 30 05:30:22.414861 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.414658    5898 cpu_manager.go:414] "containerMap: removing container" podUID="2d8a177e-9a55-45dc-8b18-548547e72b6b" containerName="cleanup"
Oct 30 05:30:22.414861 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.414666    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d8a177e-9a55-45dc-8b18-548547e72b6b" containerName="cleanup"
Oct 30 05:30:22.414861 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.414678    5898 cpu_manager.go:414] "containerMap: removing container" podUID="c687182c-2e85-4e99-8c2f-e8c972e37bcd" containerName="cleanup"
Oct 30 05:30:22.414861 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.414686    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c687182c-2e85-4e99-8c2f-e8c972e37bcd" containerName="cleanup"
Oct 30 05:30:22.414861 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.414697    5898 cpu_manager.go:414] "containerMap: removing container" podUID="7dd199d3-31ed-41c5-8570-f6a1bc1cbdc9" containerName="cleanup"
Oct 30 05:30:22.414861 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.414704    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dd199d3-31ed-41c5-8570-f6a1bc1cbdc9" containerName="cleanup"
Oct 30 05:30:22.414861 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.414735    5898 cpu_manager.go:414] "containerMap: removing container" podUID="b49de578-c4c0-4ae7-a090-4b9d8695b62c" containerName="cleanup"
Oct 30 05:30:22.414861 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.414741    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b49de578-c4c0-4ae7-a090-4b9d8695b62c" containerName="cleanup"
Oct 30 05:30:22.414861 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.414749    5898 cpu_manager.go:414] "containerMap: removing container" podUID="b49de578-c4c0-4ae7-a090-4b9d8695b62c" containerName="cleanup"
Oct 30 05:30:22.414861 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.414756    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b49de578-c4c0-4ae7-a090-4b9d8695b62c" containerName="cleanup"
Oct 30 05:30:22.414861 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.414764    5898 cpu_manager.go:414] "containerMap: removing container" podUID="fbcc00ce-d37e-49fc-8755-7100bdfdd954" containerName="cleanup"
Oct 30 05:30:22.414861 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.414771    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbcc00ce-d37e-49fc-8755-7100bdfdd954" containerName="cleanup"
Oct 30 05:30:22.414861 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.414802    5898 cpu_manager.go:414] "containerMap: removing container" podUID="3ea1e4a0-ae9b-4a99-bf48-8f5a96e33e3b" containerName="cleanup"
Oct 30 05:30:22.414861 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.414810    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ea1e4a0-ae9b-4a99-bf48-8f5a96e33e3b" containerName="cleanup"
Oct 30 05:30:22.414861 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.414818    5898 cpu_manager.go:414] "containerMap: removing container" podUID="0f696384-5b18-4ee5-8956-a4e09dfd54f2" containerName="cleanup"
Oct 30 05:30:22.414861 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.414824    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f696384-5b18-4ee5-8956-a4e09dfd54f2" containerName="cleanup"
Oct 30 05:30:22.414861 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.414833    5898 cpu_manager.go:414] "containerMap: removing container" podUID="c687182c-2e85-4e99-8c2f-e8c972e37bcd" containerName="cleanup"
Oct 30 05:30:22.414861 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.414839    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c687182c-2e85-4e99-8c2f-e8c972e37bcd" containerName="cleanup"
Oct 30 05:30:22.414861 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.414860    5898 cpu_manager.go:414] "containerMap: removing container" podUID="c687182c-2e85-4e99-8c2f-e8c972e37bcd" containerName="cleanup"
Oct 30 05:30:22.414861 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.414867    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c687182c-2e85-4e99-8c2f-e8c972e37bcd" containerName="cleanup"
Oct 30 05:30:22.416108 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.414888    5898 cpu_manager.go:414] "containerMap: removing container" podUID="0f696384-5b18-4ee5-8956-a4e09dfd54f2" containerName="cleanup"
Oct 30 05:30:22.416108 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.414894    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f696384-5b18-4ee5-8956-a4e09dfd54f2" containerName="cleanup"
Oct 30 05:30:22.416108 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.414929    5898 cpu_manager.go:414] "containerMap: removing container" podUID="2d8a177e-9a55-45dc-8b18-548547e72b6b" containerName="cleanup"
Oct 30 05:30:22.416108 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.414937    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d8a177e-9a55-45dc-8b18-548547e72b6b" containerName="cleanup"
Oct 30 05:30:22.416108 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.414956    5898 cpu_manager.go:414] "containerMap: removing container" podUID="3ea1e4a0-ae9b-4a99-bf48-8f5a96e33e3b" containerName="cleanup"
Oct 30 05:30:22.416108 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.414962    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ea1e4a0-ae9b-4a99-bf48-8f5a96e33e3b" containerName="cleanup"
Oct 30 05:30:22.416108 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.414981    5898 cpu_manager.go:414] "containerMap: removing container" podUID="0f696384-5b18-4ee5-8956-a4e09dfd54f2" containerName="cleanup"
Oct 30 05:30:22.416108 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.414988    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f696384-5b18-4ee5-8956-a4e09dfd54f2" containerName="cleanup"
Oct 30 05:30:22.416108 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.415003    5898 cpu_manager.go:414] "containerMap: removing container" podUID="b49de578-c4c0-4ae7-a090-4b9d8695b62c" containerName="cleanup"
Oct 30 05:30:22.416108 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.415010    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b49de578-c4c0-4ae7-a090-4b9d8695b62c" containerName="cleanup"
Oct 30 05:30:22.416108 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.415026    5898 cpu_manager.go:414] "containerMap: removing container" podUID="7dd199d3-31ed-41c5-8570-f6a1bc1cbdc9" containerName="cleanup"
Oct 30 05:30:22.416108 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.415032    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dd199d3-31ed-41c5-8570-f6a1bc1cbdc9" containerName="cleanup"
Oct 30 05:30:22.416108 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.415204    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ea1e4a0-ae9b-4a99-bf48-8f5a96e33e3b" containerName="cleanup"
Oct 30 05:30:22.416108 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.415217    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f696384-5b18-4ee5-8956-a4e09dfd54f2" containerName="cleanup"
Oct 30 05:30:22.416108 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.415225    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="c687182c-2e85-4e99-8c2f-e8c972e37bcd" containerName="cleanup"
Oct 30 05:30:22.416108 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.415236    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ea1e4a0-ae9b-4a99-bf48-8f5a96e33e3b" containerName="cleanup"
Oct 30 05:30:22.416108 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.415244    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="c687182c-2e85-4e99-8c2f-e8c972e37bcd" containerName="cleanup"
Oct 30 05:30:22.416108 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.415256    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f696384-5b18-4ee5-8956-a4e09dfd54f2" containerName="cleanup"
Oct 30 05:30:22.416108 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.415263    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d8a177e-9a55-45dc-8b18-548547e72b6b" containerName="cleanup"
Oct 30 05:30:22.416108 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.415272    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d8a177e-9a55-45dc-8b18-548547e72b6b" containerName="cleanup"
Oct 30 05:30:22.416108 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.415283    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dd199d3-31ed-41c5-8570-f6a1bc1cbdc9" containerName="cleanup"
Oct 30 05:30:22.416108 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.415293    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f696384-5b18-4ee5-8956-a4e09dfd54f2" containerName="cleanup"
Oct 30 05:30:22.416108 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.415305    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbcc00ce-d37e-49fc-8755-7100bdfdd954" containerName="cleanup"
Oct 30 05:30:22.416108 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.415312    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="b49de578-c4c0-4ae7-a090-4b9d8695b62c" containerName="cleanup"
Oct 30 05:30:22.416108 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.415320    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dd199d3-31ed-41c5-8570-f6a1bc1cbdc9" containerName="cleanup"
Oct 30 05:30:22.416108 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.415329    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbcc00ce-d37e-49fc-8755-7100bdfdd954" containerName="cleanup"
Oct 30 05:30:22.416108 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.415339    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d8a177e-9a55-45dc-8b18-548547e72b6b" containerName="cleanup"
Oct 30 05:30:22.416108 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.415346    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dd199d3-31ed-41c5-8570-f6a1bc1cbdc9" containerName="cleanup"
Oct 30 05:30:22.416108 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.415358    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="b49de578-c4c0-4ae7-a090-4b9d8695b62c" containerName="cleanup"
Oct 30 05:30:22.416108 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.415365    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="b49de578-c4c0-4ae7-a090-4b9d8695b62c" containerName="cleanup"
Oct 30 05:30:22.416108 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.415375    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbcc00ce-d37e-49fc-8755-7100bdfdd954" containerName="cleanup"
Oct 30 05:30:22.422647 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.422621    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b"
Oct 30 05:30:22.422779 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.422757    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b"]
Oct 30 05:30:22.460066 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.460033    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbcc00ce-d37e-49fc-8755-7100bdfdd954-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbcc00ce-d37e-49fc-8755-7100bdfdd954-maas-api-service-ca\") pod \"fbcc00ce-d37e-49fc-8755-7100bdfdd954\" (UID: \"fbcc00ce-d37e-49fc-8755-7100bdfdd954\") "
Oct 30 05:30:22.460288 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.460100    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b49de578-c4c0-4ae7-a090-4b9d8695b62c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b49de578-c4c0-4ae7-a090-4b9d8695b62c-maas-api-service-ca\") pod \"b49de578-c4c0-4ae7-a090-4b9d8695b62c\" (UID: \"b49de578-c4c0-4ae7-a090-4b9d8695b62c\") "
Oct 30 05:30:22.460393 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.460305    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b49de578-c4c0-4ae7-a090-4b9d8695b62c-kube-api-access-b8988\" (UniqueName: \"kubernetes.io/projected/b49de578-c4c0-4ae7-a090-4b9d8695b62c-kube-api-access-b8988\") pod \"b49de578-c4c0-4ae7-a090-4b9d8695b62c\" (UID: \"b49de578-c4c0-4ae7-a090-4b9d8695b62c\") "
Oct 30 05:30:22.460393 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.460365    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbcc00ce-d37e-49fc-8755-7100bdfdd954-kube-api-access-xlp4d\" (UniqueName: \"kubernetes.io/projected/fbcc00ce-d37e-49fc-8755-7100bdfdd954-kube-api-access-xlp4d\") pod \"fbcc00ce-d37e-49fc-8755-7100bdfdd954\" (UID: \"fbcc00ce-d37e-49fc-8755-7100bdfdd954\") "
Oct 30 05:30:22.460564 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.460500    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbcc00ce-d37e-49fc-8755-7100bdfdd954-maas-api-service-ca" pod "fbcc00ce-d37e-49fc-8755-7100bdfdd954" (UID: "fbcc00ce-d37e-49fc-8755-7100bdfdd954"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:30:22.460873 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.460850    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b49de578-c4c0-4ae7-a090-4b9d8695b62c-maas-api-service-ca" pod "b49de578-c4c0-4ae7-a090-4b9d8695b62c" (UID: "b49de578-c4c0-4ae7-a090-4b9d8695b62c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:30:22.461746 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.461639    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbcc00ce-d37e-49fc-8755-7100bdfdd954-maas-api-service-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:22.461887 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.461869    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g22rz\" (UniqueName: \"kubernetes.io/projected/7dd199d3-31ed-41c5-8570-f6a1bc1cbdc9-kube-api-access-g22rz\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:22.462105 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.462089    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b49de578-c4c0-4ae7-a090-4b9d8695b62c-maas-api-service-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:22.462941 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.462920    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b49de578-c4c0-4ae7-a090-4b9d8695b62c-kube-api-access-b8988" pod "b49de578-c4c0-4ae7-a090-4b9d8695b62c" (UID: "b49de578-c4c0-4ae7-a090-4b9d8695b62c"). InnerVolumeSpecName "kube-api-access-b8988". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:30:22.463154 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.463130    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbcc00ce-d37e-49fc-8755-7100bdfdd954-kube-api-access-xlp4d" pod "fbcc00ce-d37e-49fc-8755-7100bdfdd954" (UID: "fbcc00ce-d37e-49fc-8755-7100bdfdd954"). InnerVolumeSpecName "kube-api-access-xlp4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:30:22.497883 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.497852    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:22.566557 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.566502    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d295198-3d50-44bf-928c-18dd765718e8-istiod-ca-cert\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b\" (UID: \"5d295198-3d50-44bf-928c-18dd765718e8\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b"
Oct 30 05:30:22.594733 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.594687    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d295198-3d50-44bf-928c-18dd765718e8-istio-token\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b\" (UID: \"5d295198-3d50-44bf-928c-18dd765718e8\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b"
Oct 30 05:30:22.595074 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.595039    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d295198-3d50-44bf-928c-18dd765718e8-workload-certs\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b\" (UID: \"5d295198-3d50-44bf-928c-18dd765718e8\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b"
Oct 30 05:30:22.595319 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.595249    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d295198-3d50-44bf-928c-18dd765718e8-istio-data\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b\" (UID: \"5d295198-3d50-44bf-928c-18dd765718e8\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b"
Oct 30 05:30:22.595598 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.595575    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljt26\" (UniqueName: \"kubernetes.io/projected/5d295198-3d50-44bf-928c-18dd765718e8-kube-api-access-ljt26\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b\" (UID: \"5d295198-3d50-44bf-928c-18dd765718e8\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b"
Oct 30 05:30:22.599085 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.599058    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d295198-3d50-44bf-928c-18dd765718e8-workload-socket\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b\" (UID: \"5d295198-3d50-44bf-928c-18dd765718e8\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b"
Oct 30 05:30:22.635862 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.633094    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d295198-3d50-44bf-928c-18dd765718e8-credential-socket\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b\" (UID: \"5d295198-3d50-44bf-928c-18dd765718e8\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b"
Oct 30 05:30:22.635862 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.633190    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d295198-3d50-44bf-928c-18dd765718e8-istio-envoy\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b\" (UID: \"5d295198-3d50-44bf-928c-18dd765718e8\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b"
Oct 30 05:30:22.635862 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.633294    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d295198-3d50-44bf-928c-18dd765718e8-istio-podinfo\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b\" (UID: \"5d295198-3d50-44bf-928c-18dd765718e8\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b"
Oct 30 05:30:22.635862 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.633516    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8988\" (UniqueName: \"kubernetes.io/projected/b49de578-c4c0-4ae7-a090-4b9d8695b62c-kube-api-access-b8988\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:22.635862 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.633540    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlp4d\" (UniqueName: \"kubernetes.io/projected/fbcc00ce-d37e-49fc-8755-7100bdfdd954-kube-api-access-xlp4d\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:22.735194 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.735106    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljt26\" (UniqueName: \"kubernetes.io/projected/5d295198-3d50-44bf-928c-18dd765718e8-kube-api-access-ljt26\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b\" (UID: \"5d295198-3d50-44bf-928c-18dd765718e8\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b"
Oct 30 05:30:22.735194 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.735185    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d295198-3d50-44bf-928c-18dd765718e8-workload-socket\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b\" (UID: \"5d295198-3d50-44bf-928c-18dd765718e8\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b"
Oct 30 05:30:22.735436 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.735217    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d295198-3d50-44bf-928c-18dd765718e8-credential-socket\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b\" (UID: \"5d295198-3d50-44bf-928c-18dd765718e8\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b"
Oct 30 05:30:22.735436 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.735253    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d295198-3d50-44bf-928c-18dd765718e8-istio-envoy\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b\" (UID: \"5d295198-3d50-44bf-928c-18dd765718e8\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b"
Oct 30 05:30:22.735436 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.735320    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d295198-3d50-44bf-928c-18dd765718e8-istio-podinfo\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b\" (UID: \"5d295198-3d50-44bf-928c-18dd765718e8\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b"
Oct 30 05:30:22.735436 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.735388    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d295198-3d50-44bf-928c-18dd765718e8-istiod-ca-cert\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b\" (UID: \"5d295198-3d50-44bf-928c-18dd765718e8\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b"
Oct 30 05:30:22.735583 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.735566    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d295198-3d50-44bf-928c-18dd765718e8-istio-token\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b\" (UID: \"5d295198-3d50-44bf-928c-18dd765718e8\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b"
Oct 30 05:30:22.735644 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.735620    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d295198-3d50-44bf-928c-18dd765718e8-workload-certs\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b\" (UID: \"5d295198-3d50-44bf-928c-18dd765718e8\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b"
Oct 30 05:30:22.735709 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.735650    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d295198-3d50-44bf-928c-18dd765718e8-istio-data\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b\" (UID: \"5d295198-3d50-44bf-928c-18dd765718e8\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b"
Oct 30 05:30:22.735709 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.735683    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d295198-3d50-44bf-928c-18dd765718e8-credential-socket\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b\" (UID: \"5d295198-3d50-44bf-928c-18dd765718e8\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b"
Oct 30 05:30:22.735815 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.735704    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d295198-3d50-44bf-928c-18dd765718e8-workload-socket\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b\" (UID: \"5d295198-3d50-44bf-928c-18dd765718e8\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b"
Oct 30 05:30:22.736058 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.736035    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d295198-3d50-44bf-928c-18dd765718e8-istio-data\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b\" (UID: \"5d295198-3d50-44bf-928c-18dd765718e8\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b"
Oct 30 05:30:22.736168 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.736131    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d295198-3d50-44bf-928c-18dd765718e8-workload-certs\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b\" (UID: \"5d295198-3d50-44bf-928c-18dd765718e8\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b"
Oct 30 05:30:22.736241 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.736218    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d295198-3d50-44bf-928c-18dd765718e8-istiod-ca-cert\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b\" (UID: \"5d295198-3d50-44bf-928c-18dd765718e8\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b"
Oct 30 05:30:22.737804 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.737783    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d295198-3d50-44bf-928c-18dd765718e8-istio-envoy\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b\" (UID: \"5d295198-3d50-44bf-928c-18dd765718e8\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b"
Oct 30 05:30:22.737940 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.737922    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d295198-3d50-44bf-928c-18dd765718e8-istio-podinfo\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b\" (UID: \"5d295198-3d50-44bf-928c-18dd765718e8\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b"
Oct 30 05:30:22.756213 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.756185    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d295198-3d50-44bf-928c-18dd765718e8-istio-token\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b\" (UID: \"5d295198-3d50-44bf-928c-18dd765718e8\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b"
Oct 30 05:30:22.756300 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.756282    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljt26\" (UniqueName: \"kubernetes.io/projected/5d295198-3d50-44bf-928c-18dd765718e8-kube-api-access-ljt26\") pod \"e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b\" (UID: \"5d295198-3d50-44bf-928c-18dd765718e8\") " pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b"
Oct 30 05:30:22.861177 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.861147    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2f6fed-29888970-rvzpv"
Oct 30 05:30:22.861177 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.861152    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2f6fed-29888970-rvzpv" event={"ID":"fbcc00ce-d37e-49fc-8755-7100bdfdd954","Type":"ContainerDied","Data":"64357128e3b2deb9171e9582ac4dbf8ff45eecde584c061455cb8c1a43fffe63"}
Oct 30 05:30:22.861757 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.861240    5898 scope.go:122] "RemoveContainer" containerID="2ecd8a11ecfeb9b96301b05200236a834cbae6184a1c5a5d6a309a9c7b6cdc46"
Oct 30 05:30:22.869942 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.867511    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888970-k9zdq" event={"ID":"7dd199d3-31ed-41c5-8570-f6a1bc1cbdc9","Type":"ContainerDied","Data":"bae0890e2acc0f8571db92ccbf7749bfa09cadc7aa6eb8f815d8611f032fa886"}
Oct 30 05:30:22.869942 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.867627    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888970-k9zdq"
Oct 30 05:30:22.890034 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.872415    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888970-hsjd5" event={"ID":"b49de578-c4c0-4ae7-a090-4b9d8695b62c","Type":"ContainerDied","Data":"744486f17b1ccacd3e6c3a750b70686e57d5d1fed961401cedca189646b53394"}
Oct 30 05:30:22.890034 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.872545    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888970-hsjd5"
Oct 30 05:30:22.898951 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.891598    5898 scope.go:122] "RemoveContainer" containerID="b59df11c0c971052811a152cbfaa8804388c571600a7c2f09e21497da6bb1169"
Oct 30 05:30:22.934594 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.930444    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f696384-5b18-4ee5-8956-a4e09dfd54f2" path="/var/lib/kubelet/pods/0f696384-5b18-4ee5-8956-a4e09dfd54f2/volumes"
Oct 30 05:30:22.934594 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.931031    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d8a177e-9a55-45dc-8b18-548547e72b6b" path="/var/lib/kubelet/pods/2d8a177e-9a55-45dc-8b18-548547e72b6b/volumes"
Oct 30 05:30:22.934594 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.931539    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ea1e4a0-ae9b-4a99-bf48-8f5a96e33e3b" path="/var/lib/kubelet/pods/3ea1e4a0-ae9b-4a99-bf48-8f5a96e33e3b/volumes"
Oct 30 05:30:22.934594 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:22.933457    5898 scope.go:122] "RemoveContainer" containerID="655fd08b217c517364a08387897999ec6998889a152a1acd849e4c2ad9397dd9"
Oct 30 05:30:23.042652 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:23.042622    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b"
Oct 30 05:30:23.066603 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:23.066561    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2f6fed-29888970-rvzpv"]
Oct 30 05:30:23.094996 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:23.074108    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2f6fed-29888970-rvzpv"]
Oct 30 05:30:23.143010 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:23.140099    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888970-22ksr"
Oct 30 05:30:23.146808 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:23.146780    5898 patch_prober.go:29] interesting pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Oct 30 05:30:23.147313 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:23.146832    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp" podUID="80d5efb6-b3de-4f57-9249-614e1b13a04d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Oct 30 05:30:23.228033 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:23.227975    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888970-hsjd5"]
Oct 30 05:30:23.232118 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:23.232090    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ff4760-29888970-hsjd5"]
Oct 30 05:30:23.251113 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:23.250944    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2b36ef9-dd65-43f2-88fe-f360f18ba0fc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2b36ef9-dd65-43f2-88fe-f360f18ba0fc-maas-api-service-ca\") pod \"c2b36ef9-dd65-43f2-88fe-f360f18ba0fc\" (UID: \"c2b36ef9-dd65-43f2-88fe-f360f18ba0fc\") "
Oct 30 05:30:23.251113 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:23.251053    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2b36ef9-dd65-43f2-88fe-f360f18ba0fc-kube-api-access-wxqhk\" (UniqueName: \"kubernetes.io/projected/c2b36ef9-dd65-43f2-88fe-f360f18ba0fc-kube-api-access-wxqhk\") pod \"c2b36ef9-dd65-43f2-88fe-f360f18ba0fc\" (UID: \"c2b36ef9-dd65-43f2-88fe-f360f18ba0fc\") "
Oct 30 05:30:23.253763 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:23.253739    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2b36ef9-dd65-43f2-88fe-f360f18ba0fc-maas-api-service-ca" pod "c2b36ef9-dd65-43f2-88fe-f360f18ba0fc" (UID: "c2b36ef9-dd65-43f2-88fe-f360f18ba0fc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:30:23.260294 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:23.260267    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2b36ef9-dd65-43f2-88fe-f360f18ba0fc-kube-api-access-wxqhk" pod "c2b36ef9-dd65-43f2-88fe-f360f18ba0fc" (UID: "c2b36ef9-dd65-43f2-88fe-f360f18ba0fc"). InnerVolumeSpecName "kube-api-access-wxqhk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:30:23.262088 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:23.261967    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888970-k9zdq"]
Oct 30 05:30:23.268952 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:23.268925    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-44bc69-29888970-k9zdq"]
Oct 30 05:30:23.338473 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:23.338444    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888970-qhwpt"
Oct 30 05:30:23.340489 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:23.340470    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888970-mf2ct"
Oct 30 05:30:23.353413 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:23.353384    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08f30f4b-4298-494f-9fe3-de259a1991a0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08f30f4b-4298-494f-9fe3-de259a1991a0-maas-api-service-ca\") pod \"08f30f4b-4298-494f-9fe3-de259a1991a0\" (UID: \"08f30f4b-4298-494f-9fe3-de259a1991a0\") "
Oct 30 05:30:23.353533 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:23.353451    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08f30f4b-4298-494f-9fe3-de259a1991a0-kube-api-access-6jzd9\" (UniqueName: \"kubernetes.io/projected/08f30f4b-4298-494f-9fe3-de259a1991a0-kube-api-access-6jzd9\") pod \"08f30f4b-4298-494f-9fe3-de259a1991a0\" (UID: \"08f30f4b-4298-494f-9fe3-de259a1991a0\") "
Oct 30 05:30:23.353603 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:23.353587    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f05b65d9-9992-445b-af08-28759a4ef2e2-kube-api-access-mjvnw\" (UniqueName: \"kubernetes.io/projected/f05b65d9-9992-445b-af08-28759a4ef2e2-kube-api-access-mjvnw\") pod \"f05b65d9-9992-445b-af08-28759a4ef2e2\" (UID: \"f05b65d9-9992-445b-af08-28759a4ef2e2\") "
Oct 30 05:30:23.353734 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:23.353717    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f05b65d9-9992-445b-af08-28759a4ef2e2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f05b65d9-9992-445b-af08-28759a4ef2e2-maas-api-service-ca\") pod \"f05b65d9-9992-445b-af08-28759a4ef2e2\" (UID: \"f05b65d9-9992-445b-af08-28759a4ef2e2\") "
Oct 30 05:30:23.353801 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:23.353782    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08f30f4b-4298-494f-9fe3-de259a1991a0-maas-api-service-ca" pod "08f30f4b-4298-494f-9fe3-de259a1991a0" (UID: "08f30f4b-4298-494f-9fe3-de259a1991a0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:30:23.354178 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:23.354153    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f05b65d9-9992-445b-af08-28759a4ef2e2-maas-api-service-ca" pod "f05b65d9-9992-445b-af08-28759a4ef2e2" (UID: "f05b65d9-9992-445b-af08-28759a4ef2e2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:30:23.354438 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:23.354421    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08f30f4b-4298-494f-9fe3-de259a1991a0-maas-api-service-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:23.354438 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:23.354439    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2b36ef9-dd65-43f2-88fe-f360f18ba0fc-maas-api-service-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:23.354584 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:23.354449    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxqhk\" (UniqueName: \"kubernetes.io/projected/c2b36ef9-dd65-43f2-88fe-f360f18ba0fc-kube-api-access-wxqhk\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:23.354584 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:23.354459    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f05b65d9-9992-445b-af08-28759a4ef2e2-maas-api-service-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:23.356376 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:23.356350    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08f30f4b-4298-494f-9fe3-de259a1991a0-kube-api-access-6jzd9" pod "08f30f4b-4298-494f-9fe3-de259a1991a0" (UID: "08f30f4b-4298-494f-9fe3-de259a1991a0"). InnerVolumeSpecName "kube-api-access-6jzd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:30:23.356631 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:23.356608    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f05b65d9-9992-445b-af08-28759a4ef2e2-kube-api-access-mjvnw" pod "f05b65d9-9992-445b-af08-28759a4ef2e2" (UID: "f05b65d9-9992-445b-af08-28759a4ef2e2"). InnerVolumeSpecName "kube-api-access-mjvnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:30:23.425074 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:23.424356    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b"]
Oct 30 05:30:23.431169 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:23.431136    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:30:23.431290 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:23.431200    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:30:23.431290 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:23.431231    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:30:23.456037 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:23.455993    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjvnw\" (UniqueName: \"kubernetes.io/projected/f05b65d9-9992-445b-af08-28759a4ef2e2-kube-api-access-mjvnw\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:23.456037 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:23.456026    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jzd9\" (UniqueName: \"kubernetes.io/projected/08f30f4b-4298-494f-9fe3-de259a1991a0-kube-api-access-6jzd9\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:23.908147 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:23.908111    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888970-qhwpt"
Oct 30 05:30:23.908615 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:23.908110    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888970-qhwpt" event={"ID":"08f30f4b-4298-494f-9fe3-de259a1991a0","Type":"ContainerDied","Data":"24e81cb4b62dc8ecfb8ea29617d49b7ba498dd3596108b1b3e110e2dede6b95a"}
Oct 30 05:30:23.908615 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:23.908264    5898 scope.go:122] "RemoveContainer" containerID="961d2c947a825cc3c62379feca34274e5e434aa0c4fb49320349d8ee08395c72"
Oct 30 05:30:23.909949 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:23.909891    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888970-mf2ct"
Oct 30 05:30:23.910065 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:23.909990    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888970-mf2ct" event={"ID":"f05b65d9-9992-445b-af08-28759a4ef2e2","Type":"ContainerDied","Data":"53892149f69d7d70fa3472446e28fa13eb98bc76d914b4b11347a65043e62ea1"}
Oct 30 05:30:23.913967 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:23.913943    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b" event={"ID":"5d295198-3d50-44bf-928c-18dd765718e8","Type":"ContainerStarted","Data":"4142e64f2e09eaa5513d22a03122707f59dc1aaad9d58912ddb1304f305deca8"}
Oct 30 05:30:23.914116 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:23.914099    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b" event={"ID":"5d295198-3d50-44bf-928c-18dd765718e8","Type":"ContainerStarted","Data":"0fda2c671e7029007fd3a8a1555b8ba6cbf5917c6a4048032a06d3c005b148ab"}
Oct 30 05:30:23.917645 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:23.917623    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888970-22ksr" event={"ID":"c2b36ef9-dd65-43f2-88fe-f360f18ba0fc","Type":"ContainerDied","Data":"db7544bcf24fe8fffb15970d9b515d0e15eead05b26207751fe00d2414044cf3"}
Oct 30 05:30:23.917749 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:23.917709    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888970-22ksr"
Oct 30 05:30:23.923825 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:23.923799    5898 scope.go:122] "RemoveContainer" containerID="122582ec94f99ae39139b25506c5190bb5254d9e879d5c2677eea55cca9d523a"
Oct 30 05:30:23.936056 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:23.936034    5898 scope.go:122] "RemoveContainer" containerID="915419b423a9fe6db5162485f22b3619447f321c713ffa44e88d6b8a462d8f41"
Oct 30 05:30:24.002928 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:24.000540    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b" podStartSLOduration=3.000520769 podStartE2EDuration="3.000520769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:23.998472661 +0000 UTC m=+2753.222292564" watchObservedRunningTime="2026-10-30 05:30:24.000520769 +0000 UTC m=+2753.224340673"
Oct 30 05:30:24.045240 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:24.045200    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888970-mf2ct"]
Oct 30 05:30:24.045439 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:24.045260    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b"
Oct 30 05:30:24.047334 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:24.047302    5898 patch_prober.go:29] interesting pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Oct 30 05:30:24.047453 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:24.047359    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b" podUID="5d295198-3d50-44bf-928c-18dd765718e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Oct 30 05:30:24.053144 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:24.053114    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b21ba-29888970-mf2ct"]
Oct 30 05:30:24.107124 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:24.107092    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888970-22ksr"]
Oct 30 05:30:24.111179 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:24.111154    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-889ca9-29888970-22ksr"]
Oct 30 05:30:24.132122 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:24.132097    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888970-qhwpt"]
Oct 30 05:30:24.139675 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:24.139650    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cff4ac-29888970-qhwpt"]
Oct 30 05:30:24.144965 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:24.144890    5898 patch_prober.go:29] interesting pod/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Oct 30 05:30:24.145070 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:24.144991    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp" podUID="80d5efb6-b3de-4f57-9249-614e1b13a04d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Oct 30 05:30:24.380926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:24.377324    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:24.663949 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:24.663843    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h"]
Oct 30 05:30:24.664563 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:24.664425    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h" podUID="748d8cba-95b7-49de-8f41-d47a8876df1f" containerName="istio-proxy" containerID="cri-o://64c16e97dd25e311310e6af85491b50139b687b55f66e4372af15792acdc56b5" gracePeriod=30
Oct 30 05:30:24.672100 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:24.669506    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp"]
Oct 30 05:30:24.672100 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:24.669721    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp" podUID="80d5efb6-b3de-4f57-9249-614e1b13a04d" containerName="istio-proxy" containerID="cri-o://2425a88d4d75ad63aafca1dc1b93f9306645314872441fc127e1d6a6f2e88389" gracePeriod=30
Oct 30 05:30:24.725280 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:24.722959    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9"]
Oct 30 05:30:24.725280 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:24.723239    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9" podUID="ee1ddb31-cd93-4f89-9968-e94fd4484b6f" containerName="istio-proxy" containerID="cri-o://d607dfc259e9b9ba4df94affb54aaa3bfd37cfabbf75a7090b4ff7b6f9f03029" gracePeriod=30
Oct 30 05:30:24.735324 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:24.735289    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt"]
Oct 30 05:30:24.735855 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:24.735791    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt" podUID="3ef747b3-e680-47d5-8849-e26bd0946d15" containerName="istio-proxy" containerID="cri-o://ca11db02b22d6220f257314ecf5cecc9edbc75686993f7d086b77415449b07e0" gracePeriod=30
Oct 30 05:30:24.771455 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:24.771372    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b"]
Oct 30 05:30:24.941189 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:24.941120    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08f30f4b-4298-494f-9fe3-de259a1991a0" path="/var/lib/kubelet/pods/08f30f4b-4298-494f-9fe3-de259a1991a0/volumes"
Oct 30 05:30:24.942122 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:24.942103    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dd199d3-31ed-41c5-8570-f6a1bc1cbdc9" path="/var/lib/kubelet/pods/7dd199d3-31ed-41c5-8570-f6a1bc1cbdc9/volumes"
Oct 30 05:30:24.942653 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:24.942638    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b49de578-c4c0-4ae7-a090-4b9d8695b62c" path="/var/lib/kubelet/pods/b49de578-c4c0-4ae7-a090-4b9d8695b62c/volumes"
Oct 30 05:30:24.943575 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:24.943558    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2b36ef9-dd65-43f2-88fe-f360f18ba0fc" path="/var/lib/kubelet/pods/c2b36ef9-dd65-43f2-88fe-f360f18ba0fc/volumes"
Oct 30 05:30:24.944136 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:24.944120    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f05b65d9-9992-445b-af08-28759a4ef2e2" path="/var/lib/kubelet/pods/f05b65d9-9992-445b-af08-28759a4ef2e2/volumes"
Oct 30 05:30:24.944659 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:24.944645    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbcc00ce-d37e-49fc-8755-7100bdfdd954" path="/var/lib/kubelet/pods/fbcc00ce-d37e-49fc-8755-7100bdfdd954/volumes"
Oct 30 05:30:25.043219 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:25.043185    5898 patch_prober.go:29] interesting pod/e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Oct 30 05:30:25.043384 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:25.043285    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b" podUID="5d295198-3d50-44bf-928c-18dd765718e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Oct 30 05:30:25.948853 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:25.948813    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b" podUID="5d295198-3d50-44bf-928c-18dd765718e8" containerName="istio-proxy" containerID="cri-o://4142e64f2e09eaa5513d22a03122707f59dc1aaad9d58912ddb1304f305deca8" gracePeriod=30
Oct 30 05:30:26.249075 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:26.248990    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:26.542506 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:26.542423    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Oct 30 05:30:26.542506 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:26.542482    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9" podUID="ee1ddb31-cd93-4f89-9968-e94fd4484b6f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 05:30:26.602230 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:26.602188    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 05:30:26.602390 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:26.602258    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt" podUID="3ef747b3-e680-47d5-8849-e26bd0946d15" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 05:30:26.834646 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:30:26.834606    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:30:28.296352 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:28.296314    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:29.990920 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:29.988851    5898 generic.go:356] "Generic (PLEG): container finished" podID="3ef747b3-e680-47d5-8849-e26bd0946d15" containerID="ca11db02b22d6220f257314ecf5cecc9edbc75686993f7d086b77415449b07e0" exitCode=0
Oct 30 05:30:29.990920 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:29.988961    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt" event={"ID":"3ef747b3-e680-47d5-8849-e26bd0946d15","Type":"ContainerDied","Data":"ca11db02b22d6220f257314ecf5cecc9edbc75686993f7d086b77415449b07e0"}
Oct 30 05:30:29.991369 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:29.991072    5898 generic.go:356] "Generic (PLEG): container finished" podID="ee1ddb31-cd93-4f89-9968-e94fd4484b6f" containerID="d607dfc259e9b9ba4df94affb54aaa3bfd37cfabbf75a7090b4ff7b6f9f03029" exitCode=0
Oct 30 05:30:29.991369 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:29.991299    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9" event={"ID":"ee1ddb31-cd93-4f89-9968-e94fd4484b6f","Type":"ContainerDied","Data":"d607dfc259e9b9ba4df94affb54aaa3bfd37cfabbf75a7090b4ff7b6f9f03029"}
Oct 30 05:30:29.993845 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:29.993511    5898 generic.go:356] "Generic (PLEG): container finished" podID="748d8cba-95b7-49de-8f41-d47a8876df1f" containerID="64c16e97dd25e311310e6af85491b50139b687b55f66e4372af15792acdc56b5" exitCode=0
Oct 30 05:30:29.993845 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:29.993615    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h" event={"ID":"748d8cba-95b7-49de-8f41-d47a8876df1f","Type":"ContainerDied","Data":"64c16e97dd25e311310e6af85491b50139b687b55f66e4372af15792acdc56b5"}
Oct 30 05:30:30.106057 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.104298    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h"
Oct 30 05:30:30.146595 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.146554    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/748d8cba-95b7-49de-8f41-d47a8876df1f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/748d8cba-95b7-49de-8f41-d47a8876df1f-istiod-ca-cert\") pod \"748d8cba-95b7-49de-8f41-d47a8876df1f\" (UID: \"748d8cba-95b7-49de-8f41-d47a8876df1f\") "
Oct 30 05:30:30.146787 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.146636    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/748d8cba-95b7-49de-8f41-d47a8876df1f-istio-token\" (UniqueName: \"kubernetes.io/projected/748d8cba-95b7-49de-8f41-d47a8876df1f-istio-token\") pod \"748d8cba-95b7-49de-8f41-d47a8876df1f\" (UID: \"748d8cba-95b7-49de-8f41-d47a8876df1f\") "
Oct 30 05:30:30.146787 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.146695    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/748d8cba-95b7-49de-8f41-d47a8876df1f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/748d8cba-95b7-49de-8f41-d47a8876df1f-istio-podinfo\") pod \"748d8cba-95b7-49de-8f41-d47a8876df1f\" (UID: \"748d8cba-95b7-49de-8f41-d47a8876df1f\") "
Oct 30 05:30:30.146787 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.146759    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/748d8cba-95b7-49de-8f41-d47a8876df1f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/748d8cba-95b7-49de-8f41-d47a8876df1f-workload-certs\") pod \"748d8cba-95b7-49de-8f41-d47a8876df1f\" (UID: \"748d8cba-95b7-49de-8f41-d47a8876df1f\") "
Oct 30 05:30:30.146975 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.146835    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/748d8cba-95b7-49de-8f41-d47a8876df1f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/748d8cba-95b7-49de-8f41-d47a8876df1f-credential-socket\") pod \"748d8cba-95b7-49de-8f41-d47a8876df1f\" (UID: \"748d8cba-95b7-49de-8f41-d47a8876df1f\") "
Oct 30 05:30:30.147028 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.146992    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/748d8cba-95b7-49de-8f41-d47a8876df1f-kube-api-access-qgpn9\" (UniqueName: \"kubernetes.io/projected/748d8cba-95b7-49de-8f41-d47a8876df1f-kube-api-access-qgpn9\") pod \"748d8cba-95b7-49de-8f41-d47a8876df1f\" (UID: \"748d8cba-95b7-49de-8f41-d47a8876df1f\") "
Oct 30 05:30:30.147116 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.147099    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/748d8cba-95b7-49de-8f41-d47a8876df1f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/748d8cba-95b7-49de-8f41-d47a8876df1f-istio-data\") pod \"748d8cba-95b7-49de-8f41-d47a8876df1f\" (UID: \"748d8cba-95b7-49de-8f41-d47a8876df1f\") "
Oct 30 05:30:30.147171 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.147140    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/748d8cba-95b7-49de-8f41-d47a8876df1f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/748d8cba-95b7-49de-8f41-d47a8876df1f-workload-socket\") pod \"748d8cba-95b7-49de-8f41-d47a8876df1f\" (UID: \"748d8cba-95b7-49de-8f41-d47a8876df1f\") "
Oct 30 05:30:30.147232 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.147220    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/748d8cba-95b7-49de-8f41-d47a8876df1f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/748d8cba-95b7-49de-8f41-d47a8876df1f-istio-envoy\") pod \"748d8cba-95b7-49de-8f41-d47a8876df1f\" (UID: \"748d8cba-95b7-49de-8f41-d47a8876df1f\") "
Oct 30 05:30:30.149640 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.149606    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/748d8cba-95b7-49de-8f41-d47a8876df1f-istio-data" pod "748d8cba-95b7-49de-8f41-d47a8876df1f" (UID: "748d8cba-95b7-49de-8f41-d47a8876df1f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:30:30.150406 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.149808    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/748d8cba-95b7-49de-8f41-d47a8876df1f-credential-socket" pod "748d8cba-95b7-49de-8f41-d47a8876df1f" (UID: "748d8cba-95b7-49de-8f41-d47a8876df1f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:30:30.150406 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.150042    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/748d8cba-95b7-49de-8f41-d47a8876df1f-istiod-ca-cert" pod "748d8cba-95b7-49de-8f41-d47a8876df1f" (UID: "748d8cba-95b7-49de-8f41-d47a8876df1f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:30:30.150406 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.150351    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/748d8cba-95b7-49de-8f41-d47a8876df1f-workload-socket" pod "748d8cba-95b7-49de-8f41-d47a8876df1f" (UID: "748d8cba-95b7-49de-8f41-d47a8876df1f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:30:30.150723 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.150693    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/748d8cba-95b7-49de-8f41-d47a8876df1f-workload-certs" pod "748d8cba-95b7-49de-8f41-d47a8876df1f" (UID: "748d8cba-95b7-49de-8f41-d47a8876df1f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:30:30.151160 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.151046    5898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/748d8cba-95b7-49de-8f41-d47a8876df1f-istio-data\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:30.152089 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.151981    5898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/748d8cba-95b7-49de-8f41-d47a8876df1f-workload-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:30.152089 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.152004    5898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/748d8cba-95b7-49de-8f41-d47a8876df1f-istiod-ca-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:30.152089 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.152019    5898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/748d8cba-95b7-49de-8f41-d47a8876df1f-workload-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:30.152089 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.152051    5898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/748d8cba-95b7-49de-8f41-d47a8876df1f-credential-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:30.155734 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.155642    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/748d8cba-95b7-49de-8f41-d47a8876df1f-kube-api-access-qgpn9" pod "748d8cba-95b7-49de-8f41-d47a8876df1f" (UID: "748d8cba-95b7-49de-8f41-d47a8876df1f"). InnerVolumeSpecName "kube-api-access-qgpn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:30:30.156218 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.156162    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/748d8cba-95b7-49de-8f41-d47a8876df1f-istio-token" pod "748d8cba-95b7-49de-8f41-d47a8876df1f" (UID: "748d8cba-95b7-49de-8f41-d47a8876df1f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:30:30.157371 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.157349    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/748d8cba-95b7-49de-8f41-d47a8876df1f-istio-envoy" pod "748d8cba-95b7-49de-8f41-d47a8876df1f" (UID: "748d8cba-95b7-49de-8f41-d47a8876df1f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:30:30.158524 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.158487    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/748d8cba-95b7-49de-8f41-d47a8876df1f-istio-podinfo" pod "748d8cba-95b7-49de-8f41-d47a8876df1f" (UID: "748d8cba-95b7-49de-8f41-d47a8876df1f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:30:30.199015 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.198888    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:30.233752 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.233723    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt"
Oct 30 05:30:30.236162 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.236141    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9"
Oct 30 05:30:30.252932 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.252872    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee1ddb31-cd93-4f89-9968-e94fd4484b6f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee1ddb31-cd93-4f89-9968-e94fd4484b6f-istio-podinfo\") pod \"ee1ddb31-cd93-4f89-9968-e94fd4484b6f\" (UID: \"ee1ddb31-cd93-4f89-9968-e94fd4484b6f\") "
Oct 30 05:30:30.253070 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.252951    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ef747b3-e680-47d5-8849-e26bd0946d15-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ef747b3-e680-47d5-8849-e26bd0946d15-workload-certs\") pod \"3ef747b3-e680-47d5-8849-e26bd0946d15\" (UID: \"3ef747b3-e680-47d5-8849-e26bd0946d15\") "
Oct 30 05:30:30.253070 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.252997    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee1ddb31-cd93-4f89-9968-e94fd4484b6f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee1ddb31-cd93-4f89-9968-e94fd4484b6f-istio-data\") pod \"ee1ddb31-cd93-4f89-9968-e94fd4484b6f\" (UID: \"ee1ddb31-cd93-4f89-9968-e94fd4484b6f\") "
Oct 30 05:30:30.253070 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.253045    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ef747b3-e680-47d5-8849-e26bd0946d15-istio-token\" (UniqueName: \"kubernetes.io/projected/3ef747b3-e680-47d5-8849-e26bd0946d15-istio-token\") pod \"3ef747b3-e680-47d5-8849-e26bd0946d15\" (UID: \"3ef747b3-e680-47d5-8849-e26bd0946d15\") "
Oct 30 05:30:30.253257 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.253140    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee1ddb31-cd93-4f89-9968-e94fd4484b6f-istio-token\" (UniqueName: \"kubernetes.io/projected/ee1ddb31-cd93-4f89-9968-e94fd4484b6f-istio-token\") pod \"ee1ddb31-cd93-4f89-9968-e94fd4484b6f\" (UID: \"ee1ddb31-cd93-4f89-9968-e94fd4484b6f\") "
Oct 30 05:30:30.253257 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.253168    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee1ddb31-cd93-4f89-9968-e94fd4484b6f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee1ddb31-cd93-4f89-9968-e94fd4484b6f-istio-envoy\") pod \"ee1ddb31-cd93-4f89-9968-e94fd4484b6f\" (UID: \"ee1ddb31-cd93-4f89-9968-e94fd4484b6f\") "
Oct 30 05:30:30.253257 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.253226    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ef747b3-e680-47d5-8849-e26bd0946d15-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ef747b3-e680-47d5-8849-e26bd0946d15-istio-data\") pod \"3ef747b3-e680-47d5-8849-e26bd0946d15\" (UID: \"3ef747b3-e680-47d5-8849-e26bd0946d15\") "
Oct 30 05:30:30.253480 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.253345    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee1ddb31-cd93-4f89-9968-e94fd4484b6f-kube-api-access-z9qnd\" (UniqueName: \"kubernetes.io/projected/ee1ddb31-cd93-4f89-9968-e94fd4484b6f-kube-api-access-z9qnd\") pod \"ee1ddb31-cd93-4f89-9968-e94fd4484b6f\" (UID: \"ee1ddb31-cd93-4f89-9968-e94fd4484b6f\") "
Oct 30 05:30:30.253480 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.253407    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee1ddb31-cd93-4f89-9968-e94fd4484b6f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1ddb31-cd93-4f89-9968-e94fd4484b6f-credential-socket\") pod \"ee1ddb31-cd93-4f89-9968-e94fd4484b6f\" (UID: \"ee1ddb31-cd93-4f89-9968-e94fd4484b6f\") "
Oct 30 05:30:30.253480 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.253442    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3ef747b3-e680-47d5-8849-e26bd0946d15-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ef747b3-e680-47d5-8849-e26bd0946d15-istio-podinfo\") pod \"3ef747b3-e680-47d5-8849-e26bd0946d15\" (UID: \"3ef747b3-e680-47d5-8849-e26bd0946d15\") "
Oct 30 05:30:30.253635 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.253508    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ef747b3-e680-47d5-8849-e26bd0946d15-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ef747b3-e680-47d5-8849-e26bd0946d15-credential-socket\") pod \"3ef747b3-e680-47d5-8849-e26bd0946d15\" (UID: \"3ef747b3-e680-47d5-8849-e26bd0946d15\") "
Oct 30 05:30:30.253635 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.253544    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee1ddb31-cd93-4f89-9968-e94fd4484b6f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee1ddb31-cd93-4f89-9968-e94fd4484b6f-istiod-ca-cert\") pod \"ee1ddb31-cd93-4f89-9968-e94fd4484b6f\" (UID: \"ee1ddb31-cd93-4f89-9968-e94fd4484b6f\") "
Oct 30 05:30:30.253779 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.253738    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ef747b3-e680-47d5-8849-e26bd0946d15-kube-api-access-jcknz\" (UniqueName: \"kubernetes.io/projected/3ef747b3-e680-47d5-8849-e26bd0946d15-kube-api-access-jcknz\") pod \"3ef747b3-e680-47d5-8849-e26bd0946d15\" (UID: \"3ef747b3-e680-47d5-8849-e26bd0946d15\") "
Oct 30 05:30:30.253779 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.253770    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee1ddb31-cd93-4f89-9968-e94fd4484b6f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1ddb31-cd93-4f89-9968-e94fd4484b6f-workload-socket\") pod \"ee1ddb31-cd93-4f89-9968-e94fd4484b6f\" (UID: \"ee1ddb31-cd93-4f89-9968-e94fd4484b6f\") "
Oct 30 05:30:30.253884 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.253809    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ef747b3-e680-47d5-8849-e26bd0946d15-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ef747b3-e680-47d5-8849-e26bd0946d15-workload-socket\") pod \"3ef747b3-e680-47d5-8849-e26bd0946d15\" (UID: \"3ef747b3-e680-47d5-8849-e26bd0946d15\") "
Oct 30 05:30:30.253884 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.253851    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ef747b3-e680-47d5-8849-e26bd0946d15-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ef747b3-e680-47d5-8849-e26bd0946d15-istio-envoy\") pod \"3ef747b3-e680-47d5-8849-e26bd0946d15\" (UID: \"3ef747b3-e680-47d5-8849-e26bd0946d15\") "
Oct 30 05:30:30.254025 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.253884    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ef747b3-e680-47d5-8849-e26bd0946d15-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ef747b3-e680-47d5-8849-e26bd0946d15-istiod-ca-cert\") pod \"3ef747b3-e680-47d5-8849-e26bd0946d15\" (UID: \"3ef747b3-e680-47d5-8849-e26bd0946d15\") "
Oct 30 05:30:30.254025 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.253951    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee1ddb31-cd93-4f89-9968-e94fd4484b6f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee1ddb31-cd93-4f89-9968-e94fd4484b6f-workload-certs\") pod \"ee1ddb31-cd93-4f89-9968-e94fd4484b6f\" (UID: \"ee1ddb31-cd93-4f89-9968-e94fd4484b6f\") "
Oct 30 05:30:30.254932 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.254694    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ef747b3-e680-47d5-8849-e26bd0946d15-workload-socket" pod "3ef747b3-e680-47d5-8849-e26bd0946d15" (UID: "3ef747b3-e680-47d5-8849-e26bd0946d15"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:30:30.254932 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.254840    5898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/748d8cba-95b7-49de-8f41-d47a8876df1f-istio-token\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:30.254932 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.254858    5898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/748d8cba-95b7-49de-8f41-d47a8876df1f-istio-podinfo\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:30.254932 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.254872    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgpn9\" (UniqueName: \"kubernetes.io/projected/748d8cba-95b7-49de-8f41-d47a8876df1f-kube-api-access-qgpn9\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:30.255152 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.255107    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee1ddb31-cd93-4f89-9968-e94fd4484b6f-workload-certs" pod "ee1ddb31-cd93-4f89-9968-e94fd4484b6f" (UID: "ee1ddb31-cd93-4f89-9968-e94fd4484b6f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:30:30.255367 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.255198    5898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ef747b3-e680-47d5-8849-e26bd0946d15-workload-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:30.255367 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.255217    5898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/748d8cba-95b7-49de-8f41-d47a8876df1f-istio-envoy\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:30.255711 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.255487    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee1ddb31-cd93-4f89-9968-e94fd4484b6f-workload-socket" pod "ee1ddb31-cd93-4f89-9968-e94fd4484b6f" (UID: "ee1ddb31-cd93-4f89-9968-e94fd4484b6f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:30:30.256284 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.256003    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ef747b3-e680-47d5-8849-e26bd0946d15-workload-certs" pod "3ef747b3-e680-47d5-8849-e26bd0946d15" (UID: "3ef747b3-e680-47d5-8849-e26bd0946d15"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:30:30.256388 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.256281    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ef747b3-e680-47d5-8849-e26bd0946d15-credential-socket" pod "3ef747b3-e680-47d5-8849-e26bd0946d15" (UID: "3ef747b3-e680-47d5-8849-e26bd0946d15"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:30:30.256660 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.256645    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee1ddb31-cd93-4f89-9968-e94fd4484b6f-credential-socket" pod "ee1ddb31-cd93-4f89-9968-e94fd4484b6f" (UID: "ee1ddb31-cd93-4f89-9968-e94fd4484b6f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:30:30.256862 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.256849    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ef747b3-e680-47d5-8849-e26bd0946d15-istio-data" pod "3ef747b3-e680-47d5-8849-e26bd0946d15" (UID: "3ef747b3-e680-47d5-8849-e26bd0946d15"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:30:30.257190 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.257159    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee1ddb31-cd93-4f89-9968-e94fd4484b6f-istio-data" pod "ee1ddb31-cd93-4f89-9968-e94fd4484b6f" (UID: "ee1ddb31-cd93-4f89-9968-e94fd4484b6f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:30:30.257628 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.257608    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee1ddb31-cd93-4f89-9968-e94fd4484b6f-istiod-ca-cert" pod "ee1ddb31-cd93-4f89-9968-e94fd4484b6f" (UID: "ee1ddb31-cd93-4f89-9968-e94fd4484b6f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:30:30.257748 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.257668    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ef747b3-e680-47d5-8849-e26bd0946d15-istiod-ca-cert" pod "3ef747b3-e680-47d5-8849-e26bd0946d15" (UID: "3ef747b3-e680-47d5-8849-e26bd0946d15"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:30:30.262489 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.262463    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee1ddb31-cd93-4f89-9968-e94fd4484b6f-istio-token" pod "ee1ddb31-cd93-4f89-9968-e94fd4484b6f" (UID: "ee1ddb31-cd93-4f89-9968-e94fd4484b6f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:30:30.270890 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.270850    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ef747b3-e680-47d5-8849-e26bd0946d15-istio-podinfo" pod "3ef747b3-e680-47d5-8849-e26bd0946d15" (UID: "3ef747b3-e680-47d5-8849-e26bd0946d15"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:30:30.271171 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.271133    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee1ddb31-cd93-4f89-9968-e94fd4484b6f-istio-envoy" pod "ee1ddb31-cd93-4f89-9968-e94fd4484b6f" (UID: "ee1ddb31-cd93-4f89-9968-e94fd4484b6f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:30:30.271428 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.271390    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ef747b3-e680-47d5-8849-e26bd0946d15-istio-token" pod "3ef747b3-e680-47d5-8849-e26bd0946d15" (UID: "3ef747b3-e680-47d5-8849-e26bd0946d15"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:30:30.273580 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.273547    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee1ddb31-cd93-4f89-9968-e94fd4484b6f-kube-api-access-z9qnd" pod "ee1ddb31-cd93-4f89-9968-e94fd4484b6f" (UID: "ee1ddb31-cd93-4f89-9968-e94fd4484b6f"). InnerVolumeSpecName "kube-api-access-z9qnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:30:30.274196 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.274171    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee1ddb31-cd93-4f89-9968-e94fd4484b6f-istio-podinfo" pod "ee1ddb31-cd93-4f89-9968-e94fd4484b6f" (UID: "ee1ddb31-cd93-4f89-9968-e94fd4484b6f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:30:30.275031 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.275002    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ef747b3-e680-47d5-8849-e26bd0946d15-kube-api-access-jcknz" pod "3ef747b3-e680-47d5-8849-e26bd0946d15" (UID: "3ef747b3-e680-47d5-8849-e26bd0946d15"). InnerVolumeSpecName "kube-api-access-jcknz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:30:30.276612 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.276585    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ef747b3-e680-47d5-8849-e26bd0946d15-istio-envoy" pod "3ef747b3-e680-47d5-8849-e26bd0946d15" (UID: "3ef747b3-e680-47d5-8849-e26bd0946d15"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:30:30.381150 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.359642    5898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee1ddb31-cd93-4f89-9968-e94fd4484b6f-istio-token\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:30.381150 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.359680    5898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee1ddb31-cd93-4f89-9968-e94fd4484b6f-istio-envoy\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:30.381150 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.359692    5898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ef747b3-e680-47d5-8849-e26bd0946d15-istio-data\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:30.381150 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.359705    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9qnd\" (UniqueName: \"kubernetes.io/projected/ee1ddb31-cd93-4f89-9968-e94fd4484b6f-kube-api-access-z9qnd\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:30.381150 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.359719    5898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1ddb31-cd93-4f89-9968-e94fd4484b6f-credential-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:30.381150 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.359733    5898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ef747b3-e680-47d5-8849-e26bd0946d15-istio-podinfo\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:30.381150 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.359745    5898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ef747b3-e680-47d5-8849-e26bd0946d15-credential-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:30.381150 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.359757    5898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee1ddb31-cd93-4f89-9968-e94fd4484b6f-istiod-ca-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:30.381150 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.359773    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcknz\" (UniqueName: \"kubernetes.io/projected/3ef747b3-e680-47d5-8849-e26bd0946d15-kube-api-access-jcknz\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:30.381150 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.359785    5898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1ddb31-cd93-4f89-9968-e94fd4484b6f-workload-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:30.381150 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.359800    5898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ef747b3-e680-47d5-8849-e26bd0946d15-istio-envoy\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:30.381150 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.359813    5898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ef747b3-e680-47d5-8849-e26bd0946d15-istiod-ca-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:30.381150 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.359826    5898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee1ddb31-cd93-4f89-9968-e94fd4484b6f-workload-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:30.381150 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.359841    5898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee1ddb31-cd93-4f89-9968-e94fd4484b6f-istio-podinfo\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:30.381150 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.359853    5898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ef747b3-e680-47d5-8849-e26bd0946d15-workload-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:30.381150 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.359865    5898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee1ddb31-cd93-4f89-9968-e94fd4484b6f-istio-data\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:30.381150 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.359878    5898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ef747b3-e680-47d5-8849-e26bd0946d15-istio-token\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:30.676406 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.676381    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp"
Oct 30 05:30:30.766848 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.766816    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80d5efb6-b3de-4f57-9249-614e1b13a04d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80d5efb6-b3de-4f57-9249-614e1b13a04d-istiod-ca-cert\") pod \"80d5efb6-b3de-4f57-9249-614e1b13a04d\" (UID: \"80d5efb6-b3de-4f57-9249-614e1b13a04d\") "
Oct 30 05:30:30.767052 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.766864    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80d5efb6-b3de-4f57-9249-614e1b13a04d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80d5efb6-b3de-4f57-9249-614e1b13a04d-workload-certs\") pod \"80d5efb6-b3de-4f57-9249-614e1b13a04d\" (UID: \"80d5efb6-b3de-4f57-9249-614e1b13a04d\") "
Oct 30 05:30:30.767052 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.766982    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80d5efb6-b3de-4f57-9249-614e1b13a04d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d5efb6-b3de-4f57-9249-614e1b13a04d-workload-socket\") pod \"80d5efb6-b3de-4f57-9249-614e1b13a04d\" (UID: \"80d5efb6-b3de-4f57-9249-614e1b13a04d\") "
Oct 30 05:30:30.767193 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.767135    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80d5efb6-b3de-4f57-9249-614e1b13a04d-istio-token\" (UniqueName: \"kubernetes.io/projected/80d5efb6-b3de-4f57-9249-614e1b13a04d-istio-token\") pod \"80d5efb6-b3de-4f57-9249-614e1b13a04d\" (UID: \"80d5efb6-b3de-4f57-9249-614e1b13a04d\") "
Oct 30 05:30:30.767193 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.767175    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80d5efb6-b3de-4f57-9249-614e1b13a04d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80d5efb6-b3de-4f57-9249-614e1b13a04d-istio-data\") pod \"80d5efb6-b3de-4f57-9249-614e1b13a04d\" (UID: \"80d5efb6-b3de-4f57-9249-614e1b13a04d\") "
Oct 30 05:30:30.767306 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.767190    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80d5efb6-b3de-4f57-9249-614e1b13a04d-workload-certs" pod "80d5efb6-b3de-4f57-9249-614e1b13a04d" (UID: "80d5efb6-b3de-4f57-9249-614e1b13a04d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:30:30.767381 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.767310    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80d5efb6-b3de-4f57-9249-614e1b13a04d-kube-api-access-hvffk\" (UniqueName: \"kubernetes.io/projected/80d5efb6-b3de-4f57-9249-614e1b13a04d-kube-api-access-hvffk\") pod \"80d5efb6-b3de-4f57-9249-614e1b13a04d\" (UID: \"80d5efb6-b3de-4f57-9249-614e1b13a04d\") "
Oct 30 05:30:30.767444 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.767332    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80d5efb6-b3de-4f57-9249-614e1b13a04d-istiod-ca-cert" pod "80d5efb6-b3de-4f57-9249-614e1b13a04d" (UID: "80d5efb6-b3de-4f57-9249-614e1b13a04d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:30:30.767444 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.767347    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80d5efb6-b3de-4f57-9249-614e1b13a04d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80d5efb6-b3de-4f57-9249-614e1b13a04d-istio-podinfo\") pod \"80d5efb6-b3de-4f57-9249-614e1b13a04d\" (UID: \"80d5efb6-b3de-4f57-9249-614e1b13a04d\") "
Oct 30 05:30:30.767444 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.767405    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80d5efb6-b3de-4f57-9249-614e1b13a04d-istio-data" pod "80d5efb6-b3de-4f57-9249-614e1b13a04d" (UID: "80d5efb6-b3de-4f57-9249-614e1b13a04d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:30:30.767609 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.767497    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80d5efb6-b3de-4f57-9249-614e1b13a04d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80d5efb6-b3de-4f57-9249-614e1b13a04d-istio-envoy\") pod \"80d5efb6-b3de-4f57-9249-614e1b13a04d\" (UID: \"80d5efb6-b3de-4f57-9249-614e1b13a04d\") "
Oct 30 05:30:30.767609 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.767573    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80d5efb6-b3de-4f57-9249-614e1b13a04d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d5efb6-b3de-4f57-9249-614e1b13a04d-credential-socket\") pod \"80d5efb6-b3de-4f57-9249-614e1b13a04d\" (UID: \"80d5efb6-b3de-4f57-9249-614e1b13a04d\") "
Oct 30 05:30:30.768279 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.768241    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80d5efb6-b3de-4f57-9249-614e1b13a04d-credential-socket" pod "80d5efb6-b3de-4f57-9249-614e1b13a04d" (UID: "80d5efb6-b3de-4f57-9249-614e1b13a04d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:30:30.768512 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.768485    5898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d5efb6-b3de-4f57-9249-614e1b13a04d-credential-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:30.768619 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.768518    5898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80d5efb6-b3de-4f57-9249-614e1b13a04d-istiod-ca-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:30.768619 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.768533    5898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80d5efb6-b3de-4f57-9249-614e1b13a04d-workload-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:30.768619 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.768545    5898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80d5efb6-b3de-4f57-9249-614e1b13a04d-istio-data\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:30.768974 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.768950    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80d5efb6-b3de-4f57-9249-614e1b13a04d-workload-socket" pod "80d5efb6-b3de-4f57-9249-614e1b13a04d" (UID: "80d5efb6-b3de-4f57-9249-614e1b13a04d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:30:30.771321 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.771289    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80d5efb6-b3de-4f57-9249-614e1b13a04d-istio-podinfo" pod "80d5efb6-b3de-4f57-9249-614e1b13a04d" (UID: "80d5efb6-b3de-4f57-9249-614e1b13a04d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:30:30.771421 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.771400    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80d5efb6-b3de-4f57-9249-614e1b13a04d-kube-api-access-hvffk" pod "80d5efb6-b3de-4f57-9249-614e1b13a04d" (UID: "80d5efb6-b3de-4f57-9249-614e1b13a04d"). InnerVolumeSpecName "kube-api-access-hvffk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:30:30.773535 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.773507    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80d5efb6-b3de-4f57-9249-614e1b13a04d-istio-envoy" pod "80d5efb6-b3de-4f57-9249-614e1b13a04d" (UID: "80d5efb6-b3de-4f57-9249-614e1b13a04d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:30:30.776070 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.776044    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80d5efb6-b3de-4f57-9249-614e1b13a04d-istio-token" pod "80d5efb6-b3de-4f57-9249-614e1b13a04d" (UID: "80d5efb6-b3de-4f57-9249-614e1b13a04d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:30:30.870228 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.870194    5898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80d5efb6-b3de-4f57-9249-614e1b13a04d-istio-token\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:30.870228 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.870224    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvffk\" (UniqueName: \"kubernetes.io/projected/80d5efb6-b3de-4f57-9249-614e1b13a04d-kube-api-access-hvffk\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:30.870228 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.870235    5898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80d5efb6-b3de-4f57-9249-614e1b13a04d-istio-podinfo\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:30.870526 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.870243    5898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80d5efb6-b3de-4f57-9249-614e1b13a04d-istio-envoy\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:30.870526 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:30.870252    5898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d5efb6-b3de-4f57-9249-614e1b13a04d-workload-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:31.014983 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:31.014948    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h"
Oct 30 05:30:31.015466 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:31.014953    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h" event={"ID":"748d8cba-95b7-49de-8f41-d47a8876df1f","Type":"ContainerDied","Data":"c6607470134f3f971f826fb240d808336f1f922f523185bd7f1d66a6ce7068a1"}
Oct 30 05:30:31.015466 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:31.015120    5898 scope.go:122] "RemoveContainer" containerID="64c16e97dd25e311310e6af85491b50139b687b55f66e4372af15792acdc56b5"
Oct 30 05:30:31.016772 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:31.016748    5898 generic.go:356] "Generic (PLEG): container finished" podID="80d5efb6-b3de-4f57-9249-614e1b13a04d" containerID="2425a88d4d75ad63aafca1dc1b93f9306645314872441fc127e1d6a6f2e88389" exitCode=0
Oct 30 05:30:31.016878 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:31.016822    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp"
Oct 30 05:30:31.016878 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:31.016833    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp" event={"ID":"80d5efb6-b3de-4f57-9249-614e1b13a04d","Type":"ContainerDied","Data":"2425a88d4d75ad63aafca1dc1b93f9306645314872441fc127e1d6a6f2e88389"}
Oct 30 05:30:31.017020 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:31.016878    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp" event={"ID":"80d5efb6-b3de-4f57-9249-614e1b13a04d","Type":"ContainerDied","Data":"cd7ac96374efaa432e44b06869336d0ae9e0d1dc855daebbd5f6d56d4a02fe2c"}
Oct 30 05:30:31.018773 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:31.018755    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt"
Oct 30 05:30:31.018773 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:31.018759    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt" event={"ID":"3ef747b3-e680-47d5-8849-e26bd0946d15","Type":"ContainerDied","Data":"d718c606212d54458a12d09c696f8778fe3e866b14f8db9a4b7741a5ec83f19f"}
Oct 30 05:30:31.021451 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:31.021432    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9" event={"ID":"ee1ddb31-cd93-4f89-9968-e94fd4484b6f","Type":"ContainerDied","Data":"ecc704c72f08a62854b67c91f9171645a27f8f9fe1a27ae7c61b1d2dd03f3670"}
Oct 30 05:30:31.021559 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:31.021510    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9"
Oct 30 05:30:31.034683 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:31.034661    5898 scope.go:122] "RemoveContainer" containerID="2425a88d4d75ad63aafca1dc1b93f9306645314872441fc127e1d6a6f2e88389"
Oct 30 05:30:31.045828 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:31.045807    5898 scope.go:122] "RemoveContainer" containerID="2425a88d4d75ad63aafca1dc1b93f9306645314872441fc127e1d6a6f2e88389"
Oct 30 05:30:31.046160 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:30:31.046133    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2425a88d4d75ad63aafca1dc1b93f9306645314872441fc127e1d6a6f2e88389\": container with ID starting with 2425a88d4d75ad63aafca1dc1b93f9306645314872441fc127e1d6a6f2e88389 not found: ID does not exist" containerID="2425a88d4d75ad63aafca1dc1b93f9306645314872441fc127e1d6a6f2e88389"
Oct 30 05:30:31.046246 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:31.046172    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2425a88d4d75ad63aafca1dc1b93f9306645314872441fc127e1d6a6f2e88389"} err="failed to get container status \"2425a88d4d75ad63aafca1dc1b93f9306645314872441fc127e1d6a6f2e88389\": rpc error: code = NotFound desc = could not find container \"2425a88d4d75ad63aafca1dc1b93f9306645314872441fc127e1d6a6f2e88389\": container with ID starting with 2425a88d4d75ad63aafca1dc1b93f9306645314872441fc127e1d6a6f2e88389 not found: ID does not exist"
Oct 30 05:30:31.046246 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:31.046195    5898 scope.go:122] "RemoveContainer" containerID="ca11db02b22d6220f257314ecf5cecc9edbc75686993f7d086b77415449b07e0"
Oct 30 05:30:31.057111 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:31.057090    5898 scope.go:122] "RemoveContainer" containerID="d607dfc259e9b9ba4df94affb54aaa3bfd37cfabbf75a7090b4ff7b6f9f03029"
Oct 30 05:30:31.072061 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:31.072038    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9"]
Oct 30 05:30:31.083254 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:31.083229    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-sv6t9"]
Oct 30 05:30:31.110068 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:31.110044    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp"]
Oct 30 05:30:31.117624 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:31.117600    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-2d21ab-openshift-default-7d7974c69f-5nbzp"]
Oct 30 05:30:31.151909 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:31.151878    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt"]
Oct 30 05:30:31.155990 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:31.155963    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-dpgjt"]
Oct 30 05:30:31.191261 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:31.191232    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h"]
Oct 30 05:30:31.197409 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:31.197384    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0dba31-openshift-default-7f4847f8cc-lnp9h"]
Oct 30 05:30:31.952340 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:31.952309    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:32.893497 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:32.893465    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ef747b3-e680-47d5-8849-e26bd0946d15" path="/var/lib/kubelet/pods/3ef747b3-e680-47d5-8849-e26bd0946d15/volumes"
Oct 30 05:30:32.894071 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:32.894054    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="748d8cba-95b7-49de-8f41-d47a8876df1f" path="/var/lib/kubelet/pods/748d8cba-95b7-49de-8f41-d47a8876df1f/volumes"
Oct 30 05:30:32.894552 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:32.894538    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80d5efb6-b3de-4f57-9249-614e1b13a04d" path="/var/lib/kubelet/pods/80d5efb6-b3de-4f57-9249-614e1b13a04d/volumes"
Oct 30 05:30:32.895320 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:32.895303    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee1ddb31-cd93-4f89-9968-e94fd4484b6f" path="/var/lib/kubelet/pods/ee1ddb31-cd93-4f89-9968-e94fd4484b6f/volumes"
Oct 30 05:30:33.847688 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:33.847654    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:35.665542 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.665097    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:35.796761 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.796689    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b"
Oct 30 05:30:35.872981 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.872949    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bx4nb"]
Oct 30 05:30:35.873610 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.873596    5898 cpu_manager.go:414] "containerMap: removing container" podUID="08f30f4b-4298-494f-9fe3-de259a1991a0" containerName="cleanup"
Oct 30 05:30:35.873652 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.873613    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08f30f4b-4298-494f-9fe3-de259a1991a0" containerName="cleanup"
Oct 30 05:30:35.873652 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.873623    5898 cpu_manager.go:414] "containerMap: removing container" podUID="ee1ddb31-cd93-4f89-9968-e94fd4484b6f" containerName="istio-proxy"
Oct 30 05:30:35.873652 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.873628    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee1ddb31-cd93-4f89-9968-e94fd4484b6f" containerName="istio-proxy"
Oct 30 05:30:35.873652 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.873637    5898 cpu_manager.go:414] "containerMap: removing container" podUID="c2b36ef9-dd65-43f2-88fe-f360f18ba0fc" containerName="cleanup"
Oct 30 05:30:35.873652 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.873644    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2b36ef9-dd65-43f2-88fe-f360f18ba0fc" containerName="cleanup"
Oct 30 05:30:35.873809 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.873655    5898 cpu_manager.go:414] "containerMap: removing container" podUID="f05b65d9-9992-445b-af08-28759a4ef2e2" containerName="cleanup"
Oct 30 05:30:35.873809 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.873661    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f05b65d9-9992-445b-af08-28759a4ef2e2" containerName="cleanup"
Oct 30 05:30:35.873809 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.873668    5898 cpu_manager.go:414] "containerMap: removing container" podUID="3ea1e4a0-ae9b-4a99-bf48-8f5a96e33e3b" containerName="cleanup"
Oct 30 05:30:35.873809 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.873673    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ea1e4a0-ae9b-4a99-bf48-8f5a96e33e3b" containerName="cleanup"
Oct 30 05:30:35.873809 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.873683    5898 cpu_manager.go:414] "containerMap: removing container" podUID="f05b65d9-9992-445b-af08-28759a4ef2e2" containerName="cleanup"
Oct 30 05:30:35.873809 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.873687    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f05b65d9-9992-445b-af08-28759a4ef2e2" containerName="cleanup"
Oct 30 05:30:35.873809 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.873702    5898 cpu_manager.go:414] "containerMap: removing container" podUID="80d5efb6-b3de-4f57-9249-614e1b13a04d" containerName="istio-proxy"
Oct 30 05:30:35.873809 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.873706    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80d5efb6-b3de-4f57-9249-614e1b13a04d" containerName="istio-proxy"
Oct 30 05:30:35.873809 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.873714    5898 cpu_manager.go:414] "containerMap: removing container" podUID="c2b36ef9-dd65-43f2-88fe-f360f18ba0fc" containerName="cleanup"
Oct 30 05:30:35.873809 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.873719    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2b36ef9-dd65-43f2-88fe-f360f18ba0fc" containerName="cleanup"
Oct 30 05:30:35.873809 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.873725    5898 cpu_manager.go:414] "containerMap: removing container" podUID="2d8a177e-9a55-45dc-8b18-548547e72b6b" containerName="cleanup"
Oct 30 05:30:35.873809 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.873729    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d8a177e-9a55-45dc-8b18-548547e72b6b" containerName="cleanup"
Oct 30 05:30:35.873809 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.873735    5898 cpu_manager.go:414] "containerMap: removing container" podUID="7dd199d3-31ed-41c5-8570-f6a1bc1cbdc9" containerName="cleanup"
Oct 30 05:30:35.873809 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.873739    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dd199d3-31ed-41c5-8570-f6a1bc1cbdc9" containerName="cleanup"
Oct 30 05:30:35.873809 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.873746    5898 cpu_manager.go:414] "containerMap: removing container" podUID="748d8cba-95b7-49de-8f41-d47a8876df1f" containerName="istio-proxy"
Oct 30 05:30:35.873809 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.873750    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="748d8cba-95b7-49de-8f41-d47a8876df1f" containerName="istio-proxy"
Oct 30 05:30:35.873809 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.873761    5898 cpu_manager.go:414] "containerMap: removing container" podUID="3ef747b3-e680-47d5-8849-e26bd0946d15" containerName="istio-proxy"
Oct 30 05:30:35.873809 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.873765    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ef747b3-e680-47d5-8849-e26bd0946d15" containerName="istio-proxy"
Oct 30 05:30:35.873809 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.873777    5898 cpu_manager.go:414] "containerMap: removing container" podUID="08f30f4b-4298-494f-9fe3-de259a1991a0" containerName="cleanup"
Oct 30 05:30:35.873809 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.873782    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08f30f4b-4298-494f-9fe3-de259a1991a0" containerName="cleanup"
Oct 30 05:30:35.873809 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.873789    5898 cpu_manager.go:414] "containerMap: removing container" podUID="5d295198-3d50-44bf-928c-18dd765718e8" containerName="istio-proxy"
Oct 30 05:30:35.873809 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.873793    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d295198-3d50-44bf-928c-18dd765718e8" containerName="istio-proxy"
Oct 30 05:30:35.873809 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.873799    5898 cpu_manager.go:414] "containerMap: removing container" podUID="fbcc00ce-d37e-49fc-8755-7100bdfdd954" containerName="cleanup"
Oct 30 05:30:35.873809 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.873803    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbcc00ce-d37e-49fc-8755-7100bdfdd954" containerName="cleanup"
Oct 30 05:30:35.873809 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.873811    5898 cpu_manager.go:414] "containerMap: removing container" podUID="08f30f4b-4298-494f-9fe3-de259a1991a0" containerName="cleanup"
Oct 30 05:30:35.873809 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.873815    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08f30f4b-4298-494f-9fe3-de259a1991a0" containerName="cleanup"
Oct 30 05:30:35.874623 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.873827    5898 cpu_manager.go:414] "containerMap: removing container" podUID="c2b36ef9-dd65-43f2-88fe-f360f18ba0fc" containerName="cleanup"
Oct 30 05:30:35.874623 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.873832    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2b36ef9-dd65-43f2-88fe-f360f18ba0fc" containerName="cleanup"
Oct 30 05:30:35.874623 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.873950    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="80d5efb6-b3de-4f57-9249-614e1b13a04d" containerName="istio-proxy"
Oct 30 05:30:35.874623 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.873960    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2b36ef9-dd65-43f2-88fe-f360f18ba0fc" containerName="cleanup"
Oct 30 05:30:35.874623 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.873967    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="748d8cba-95b7-49de-8f41-d47a8876df1f" containerName="istio-proxy"
Oct 30 05:30:35.874623 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.873972    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ef747b3-e680-47d5-8849-e26bd0946d15" containerName="istio-proxy"
Oct 30 05:30:35.874623 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.873979    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="f05b65d9-9992-445b-af08-28759a4ef2e2" containerName="cleanup"
Oct 30 05:30:35.874623 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.873986    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="08f30f4b-4298-494f-9fe3-de259a1991a0" containerName="cleanup"
Oct 30 05:30:35.874623 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.873991    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d295198-3d50-44bf-928c-18dd765718e8" containerName="istio-proxy"
Oct 30 05:30:35.874623 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.873997    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="08f30f4b-4298-494f-9fe3-de259a1991a0" containerName="cleanup"
Oct 30 05:30:35.874623 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.874005    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ea1e4a0-ae9b-4a99-bf48-8f5a96e33e3b" containerName="cleanup"
Oct 30 05:30:35.874623 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.874015    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="c687182c-2e85-4e99-8c2f-e8c972e37bcd" containerName="cleanup"
Oct 30 05:30:35.874623 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.874026    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="08f30f4b-4298-494f-9fe3-de259a1991a0" containerName="cleanup"
Oct 30 05:30:35.874623 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.874034    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee1ddb31-cd93-4f89-9968-e94fd4484b6f" containerName="istio-proxy"
Oct 30 05:30:35.874623 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.874043    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2b36ef9-dd65-43f2-88fe-f360f18ba0fc" containerName="cleanup"
Oct 30 05:30:35.874623 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.874052    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="f05b65d9-9992-445b-af08-28759a4ef2e2" containerName="cleanup"
Oct 30 05:30:35.875225 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.875202    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d295198-3d50-44bf-928c-18dd765718e8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d295198-3d50-44bf-928c-18dd765718e8-istiod-ca-cert\") pod \"5d295198-3d50-44bf-928c-18dd765718e8\" (UID: \"5d295198-3d50-44bf-928c-18dd765718e8\") "
Oct 30 05:30:35.875336 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.875319    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d295198-3d50-44bf-928c-18dd765718e8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d295198-3d50-44bf-928c-18dd765718e8-istio-podinfo\") pod \"5d295198-3d50-44bf-928c-18dd765718e8\" (UID: \"5d295198-3d50-44bf-928c-18dd765718e8\") "
Oct 30 05:30:35.875398 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.875358    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d295198-3d50-44bf-928c-18dd765718e8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d295198-3d50-44bf-928c-18dd765718e8-credential-socket\") pod \"5d295198-3d50-44bf-928c-18dd765718e8\" (UID: \"5d295198-3d50-44bf-928c-18dd765718e8\") "
Oct 30 05:30:35.875451 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.875402    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d295198-3d50-44bf-928c-18dd765718e8-kube-api-access-ljt26\" (UniqueName: \"kubernetes.io/projected/5d295198-3d50-44bf-928c-18dd765718e8-kube-api-access-ljt26\") pod \"5d295198-3d50-44bf-928c-18dd765718e8\" (UID: \"5d295198-3d50-44bf-928c-18dd765718e8\") "
Oct 30 05:30:35.875504 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.875466    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d295198-3d50-44bf-928c-18dd765718e8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d295198-3d50-44bf-928c-18dd765718e8-workload-socket\") pod \"5d295198-3d50-44bf-928c-18dd765718e8\" (UID: \"5d295198-3d50-44bf-928c-18dd765718e8\") "
Oct 30 05:30:35.875577 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.875512    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d295198-3d50-44bf-928c-18dd765718e8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d295198-3d50-44bf-928c-18dd765718e8-istio-envoy\") pod \"5d295198-3d50-44bf-928c-18dd765718e8\" (UID: \"5d295198-3d50-44bf-928c-18dd765718e8\") "
Oct 30 05:30:35.875669 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.875645    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d295198-3d50-44bf-928c-18dd765718e8-istiod-ca-cert" pod "5d295198-3d50-44bf-928c-18dd765718e8" (UID: "5d295198-3d50-44bf-928c-18dd765718e8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:30:35.875730 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.875701    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d295198-3d50-44bf-928c-18dd765718e8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d295198-3d50-44bf-928c-18dd765718e8-istio-data\") pod \"5d295198-3d50-44bf-928c-18dd765718e8\" (UID: \"5d295198-3d50-44bf-928c-18dd765718e8\") "
Oct 30 05:30:35.875785 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.875770    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d295198-3d50-44bf-928c-18dd765718e8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d295198-3d50-44bf-928c-18dd765718e8-workload-certs\") pod \"5d295198-3d50-44bf-928c-18dd765718e8\" (UID: \"5d295198-3d50-44bf-928c-18dd765718e8\") "
Oct 30 05:30:35.875879 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.875863    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d295198-3d50-44bf-928c-18dd765718e8-istio-token\" (UniqueName: \"kubernetes.io/projected/5d295198-3d50-44bf-928c-18dd765718e8-istio-token\") pod \"5d295198-3d50-44bf-928c-18dd765718e8\" (UID: \"5d295198-3d50-44bf-928c-18dd765718e8\") "
Oct 30 05:30:35.876222 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.876201    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d295198-3d50-44bf-928c-18dd765718e8-credential-socket" pod "5d295198-3d50-44bf-928c-18dd765718e8" (UID: "5d295198-3d50-44bf-928c-18dd765718e8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:30:35.876688 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.876661    5898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d295198-3d50-44bf-928c-18dd765718e8-istiod-ca-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:35.876795 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.876693    5898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d295198-3d50-44bf-928c-18dd765718e8-credential-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:35.878101 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.878078    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bx4nb"
Oct 30 05:30:35.878821 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.878760    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d295198-3d50-44bf-928c-18dd765718e8-istio-data" pod "5d295198-3d50-44bf-928c-18dd765718e8" (UID: "5d295198-3d50-44bf-928c-18dd765718e8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:30:35.878821 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.878794    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d295198-3d50-44bf-928c-18dd765718e8-istio-envoy" pod "5d295198-3d50-44bf-928c-18dd765718e8" (UID: "5d295198-3d50-44bf-928c-18dd765718e8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:30:35.879212 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.879015    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d295198-3d50-44bf-928c-18dd765718e8-istio-token" pod "5d295198-3d50-44bf-928c-18dd765718e8" (UID: "5d295198-3d50-44bf-928c-18dd765718e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:30:35.879212 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.879023    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d295198-3d50-44bf-928c-18dd765718e8-workload-socket" pod "5d295198-3d50-44bf-928c-18dd765718e8" (UID: "5d295198-3d50-44bf-928c-18dd765718e8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:30:35.879410 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.879383    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d295198-3d50-44bf-928c-18dd765718e8-workload-certs" pod "5d295198-3d50-44bf-928c-18dd765718e8" (UID: "5d295198-3d50-44bf-928c-18dd765718e8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:30:35.883260 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.883242    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d295198-3d50-44bf-928c-18dd765718e8-istio-podinfo" pod "5d295198-3d50-44bf-928c-18dd765718e8" (UID: "5d295198-3d50-44bf-928c-18dd765718e8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:30:35.885337 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.885307    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d295198-3d50-44bf-928c-18dd765718e8-kube-api-access-ljt26" pod "5d295198-3d50-44bf-928c-18dd765718e8" (UID: "5d295198-3d50-44bf-928c-18dd765718e8"). InnerVolumeSpecName "kube-api-access-ljt26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:30:35.899509 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.899474    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bx4nb"]
Oct 30 05:30:35.977782 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.977743    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhj5v\" (UniqueName: \"kubernetes.io/projected/b258ca46-73e4-403f-82f6-74e0cd489856-kube-api-access-xhj5v\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bx4nb\" (UID: \"b258ca46-73e4-403f-82f6-74e0cd489856\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bx4nb"
Oct 30 05:30:35.978078 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.977799    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b258ca46-73e4-403f-82f6-74e0cd489856-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bx4nb\" (UID: \"b258ca46-73e4-403f-82f6-74e0cd489856\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bx4nb"
Oct 30 05:30:35.978078 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.977824    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b258ca46-73e4-403f-82f6-74e0cd489856-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bx4nb\" (UID: \"b258ca46-73e4-403f-82f6-74e0cd489856\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bx4nb"
Oct 30 05:30:35.978078 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.977873    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b258ca46-73e4-403f-82f6-74e0cd489856-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bx4nb\" (UID: \"b258ca46-73e4-403f-82f6-74e0cd489856\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bx4nb"
Oct 30 05:30:35.978078 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.977976    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b258ca46-73e4-403f-82f6-74e0cd489856-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bx4nb\" (UID: \"b258ca46-73e4-403f-82f6-74e0cd489856\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bx4nb"
Oct 30 05:30:35.978078 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.978004    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b258ca46-73e4-403f-82f6-74e0cd489856-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bx4nb\" (UID: \"b258ca46-73e4-403f-82f6-74e0cd489856\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bx4nb"
Oct 30 05:30:35.978078 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.978037    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b258ca46-73e4-403f-82f6-74e0cd489856-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bx4nb\" (UID: \"b258ca46-73e4-403f-82f6-74e0cd489856\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bx4nb"
Oct 30 05:30:35.978517 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.978172    5898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d295198-3d50-44bf-928c-18dd765718e8-istio-token\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:35.978517 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.978187    5898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d295198-3d50-44bf-928c-18dd765718e8-istio-podinfo\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:35.978517 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.978201    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljt26\" (UniqueName: \"kubernetes.io/projected/5d295198-3d50-44bf-928c-18dd765718e8-kube-api-access-ljt26\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:35.978517 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.978218    5898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d295198-3d50-44bf-928c-18dd765718e8-workload-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:35.978517 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.978233    5898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d295198-3d50-44bf-928c-18dd765718e8-istio-envoy\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:35.978517 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.978241    5898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d295198-3d50-44bf-928c-18dd765718e8-istio-data\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:35.978517 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.978252    5898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d295198-3d50-44bf-928c-18dd765718e8-workload-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:30:35.983287 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:35.983258    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:36.079451 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:36.079414    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhj5v\" (UniqueName: \"kubernetes.io/projected/b258ca46-73e4-403f-82f6-74e0cd489856-kube-api-access-xhj5v\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bx4nb\" (UID: \"b258ca46-73e4-403f-82f6-74e0cd489856\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bx4nb"
Oct 30 05:30:36.079451 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:36.079456    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b258ca46-73e4-403f-82f6-74e0cd489856-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bx4nb\" (UID: \"b258ca46-73e4-403f-82f6-74e0cd489856\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bx4nb"
Oct 30 05:30:36.079699 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:36.079474    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b258ca46-73e4-403f-82f6-74e0cd489856-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bx4nb\" (UID: \"b258ca46-73e4-403f-82f6-74e0cd489856\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bx4nb"
Oct 30 05:30:36.079699 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:36.079514    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b258ca46-73e4-403f-82f6-74e0cd489856-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bx4nb\" (UID: \"b258ca46-73e4-403f-82f6-74e0cd489856\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bx4nb"
Oct 30 05:30:36.079699 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:36.079620    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b258ca46-73e4-403f-82f6-74e0cd489856-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bx4nb\" (UID: \"b258ca46-73e4-403f-82f6-74e0cd489856\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bx4nb"
Oct 30 05:30:36.079699 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:36.079649    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b258ca46-73e4-403f-82f6-74e0cd489856-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bx4nb\" (UID: \"b258ca46-73e4-403f-82f6-74e0cd489856\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bx4nb"
Oct 30 05:30:36.079699 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:36.079686    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b258ca46-73e4-403f-82f6-74e0cd489856-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bx4nb\" (UID: \"b258ca46-73e4-403f-82f6-74e0cd489856\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bx4nb"
Oct 30 05:30:36.080856 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:36.080798    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b258ca46-73e4-403f-82f6-74e0cd489856-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bx4nb\" (UID: \"b258ca46-73e4-403f-82f6-74e0cd489856\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bx4nb"
Oct 30 05:30:36.082747 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:36.082701    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b258ca46-73e4-403f-82f6-74e0cd489856-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bx4nb\" (UID: \"b258ca46-73e4-403f-82f6-74e0cd489856\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bx4nb"
Oct 30 05:30:36.082747 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:36.082722    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b258ca46-73e4-403f-82f6-74e0cd489856-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bx4nb\" (UID: \"b258ca46-73e4-403f-82f6-74e0cd489856\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bx4nb"
Oct 30 05:30:36.082941 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:36.082757    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b258ca46-73e4-403f-82f6-74e0cd489856-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bx4nb\" (UID: \"b258ca46-73e4-403f-82f6-74e0cd489856\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bx4nb"
Oct 30 05:30:36.083701 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:36.083675    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b258ca46-73e4-403f-82f6-74e0cd489856-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bx4nb\" (UID: \"b258ca46-73e4-403f-82f6-74e0cd489856\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bx4nb"
Oct 30 05:30:36.099445 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:36.099414    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b258ca46-73e4-403f-82f6-74e0cd489856-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bx4nb\" (UID: \"b258ca46-73e4-403f-82f6-74e0cd489856\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bx4nb"
Oct 30 05:30:36.099768 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:36.099747    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhj5v\" (UniqueName: \"kubernetes.io/projected/b258ca46-73e4-403f-82f6-74e0cd489856-kube-api-access-xhj5v\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bx4nb\" (UID: \"b258ca46-73e4-403f-82f6-74e0cd489856\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bx4nb"
Oct 30 05:30:36.117853 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:36.117816    5898 generic.go:356] "Generic (PLEG): container finished" podID="5d295198-3d50-44bf-928c-18dd765718e8" containerID="4142e64f2e09eaa5513d22a03122707f59dc1aaad9d58912ddb1304f305deca8" exitCode=0
Oct 30 05:30:36.118001 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:36.117864    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b"
Oct 30 05:30:36.118001 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:36.117881    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b" event={"ID":"5d295198-3d50-44bf-928c-18dd765718e8","Type":"ContainerDied","Data":"4142e64f2e09eaa5513d22a03122707f59dc1aaad9d58912ddb1304f305deca8"}
Oct 30 05:30:36.118001 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:36.117955    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b" event={"ID":"5d295198-3d50-44bf-928c-18dd765718e8","Type":"ContainerDied","Data":"0fda2c671e7029007fd3a8a1555b8ba6cbf5917c6a4048032a06d3c005b148ab"}
Oct 30 05:30:36.118001 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:36.117988    5898 scope.go:122] "RemoveContainer" containerID="4142e64f2e09eaa5513d22a03122707f59dc1aaad9d58912ddb1304f305deca8"
Oct 30 05:30:36.131459 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:36.131438    5898 scope.go:122] "RemoveContainer" containerID="4142e64f2e09eaa5513d22a03122707f59dc1aaad9d58912ddb1304f305deca8"
Oct 30 05:30:36.131836 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:30:36.131786    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4142e64f2e09eaa5513d22a03122707f59dc1aaad9d58912ddb1304f305deca8\": container with ID starting with 4142e64f2e09eaa5513d22a03122707f59dc1aaad9d58912ddb1304f305deca8 not found: ID does not exist" containerID="4142e64f2e09eaa5513d22a03122707f59dc1aaad9d58912ddb1304f305deca8"
Oct 30 05:30:36.131945 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:36.131837    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4142e64f2e09eaa5513d22a03122707f59dc1aaad9d58912ddb1304f305deca8"} err="failed to get container status \"4142e64f2e09eaa5513d22a03122707f59dc1aaad9d58912ddb1304f305deca8\": rpc error: code = NotFound desc = could not find container \"4142e64f2e09eaa5513d22a03122707f59dc1aaad9d58912ddb1304f305deca8\": container with ID starting with 4142e64f2e09eaa5513d22a03122707f59dc1aaad9d58912ddb1304f305deca8 not found: ID does not exist"
Oct 30 05:30:36.220945 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:36.218739    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b"]
Oct 30 05:30:36.283938 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:36.272734    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bx4nb"
Oct 30 05:30:36.459984 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:36.458309    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-f4b5b"]
Oct 30 05:30:36.853191 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:36.853167    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bx4nb"]
Oct 30 05:30:36.855966 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:30:36.855938    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb258ca46_73e4_403f_82f6_74e0cd489856.slice/crio-7e5c6e22a6b0b013524b83e1106c14d9b2420b0d0c829f6a12927a1cff8a77f2 WatchSource:0}: Error finding container 7e5c6e22a6b0b013524b83e1106c14d9b2420b0d0c829f6a12927a1cff8a77f2: Status 404 returned error can't find the container with id 7e5c6e22a6b0b013524b83e1106c14d9b2420b0d0c829f6a12927a1cff8a77f2
Oct 30 05:30:36.859224 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:36.859194    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:30:36.859297 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:36.859254    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:30:36.895108 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:36.895075    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d295198-3d50-44bf-928c-18dd765718e8" path="/var/lib/kubelet/pods/5d295198-3d50-44bf-928c-18dd765718e8/volumes"
Oct 30 05:30:37.128255 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:37.128163    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bx4nb" event={"ID":"b258ca46-73e4-403f-82f6-74e0cd489856","Type":"ContainerStarted","Data":"be4d849712a6bdffff9bc9a4a6716a5c4a283ac634149d85ffab97f06b5d57ac"}
Oct 30 05:30:37.128255 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:37.128237    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bx4nb"
Oct 30 05:30:37.128475 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:37.128259    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bx4nb" event={"ID":"b258ca46-73e4-403f-82f6-74e0cd489856","Type":"ContainerStarted","Data":"7e5c6e22a6b0b013524b83e1106c14d9b2420b0d0c829f6a12927a1cff8a77f2"}
Oct 30 05:30:37.129717 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:37.129686    5898 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-bx4nb container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 05:30:37.129829 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:37.129738    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bx4nb" podUID="b258ca46-73e4-403f-82f6-74e0cd489856" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 05:30:37.178944 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:37.175469    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bx4nb" podStartSLOduration=13.175452174 podStartE2EDuration="13.175452174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:37.171080416 +0000 UTC m=+2766.394900318" watchObservedRunningTime="2026-10-30 05:30:37.175452174 +0000 UTC m=+2766.399272076"
Oct 30 05:30:37.883187 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:37.883155    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:38.137938 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:38.137828    5898 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-bx4nb container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 05:30:38.137938 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:38.137887    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bx4nb" podUID="b258ca46-73e4-403f-82f6-74e0cd489856" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 05:30:40.450563 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:40.450529    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:41.138595 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:41.138565    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bx4nb"
Oct 30 05:30:42.189111 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:42.189077    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:42.954341 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:42.954306    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:44.059361 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:44.059326    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:45.630535 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:45.630500    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:45.819362 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:45.819326    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:46.857481 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:46.857445    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:48.771207 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:48.771173    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:50.512429 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:50.512374    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:51.933409 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:51.933353    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:52.343924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:52.338835    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:52.476797 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:52.472459    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:52.649831 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:52.649750    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:52.756234 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:52.756203    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:30:52.766833 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:52.766805    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:54.565968 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:54.561648    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:55.522885 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:55.522792    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:56.669324 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:56.669278    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:30:56.824548 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:30:56.823855    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:30:58.519681 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:30:58.519646    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:00.399083 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:00.399052    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:02.193302 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:02.193272    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:03.957148 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:03.957117    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:05.835555 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:05.835502    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:08.127065 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:08.127027    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:10.009196 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:10.009162    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:12.329529 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:12.329453    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:14.111265 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:14.111235    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:15.961866 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:15.961834    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:17.697093 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:17.697061    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:19.469799 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:19.469764    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:21.260770 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:21.260739    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:22.992844 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:22.992811    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:24.900460 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:24.900426    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:26.820128 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:31:26.820079    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:31:27.040186 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:27.040154    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:28.965106 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:28.965073    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:30.502416 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:30.502373    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:30.714236 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:30.714186    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:30.816828 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:30.816727    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:30.992582 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:30.992526    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:31.123838 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:31.123800    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:31.244572 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:31.244541    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:31.367560 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:31.367511    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:31.781428 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:31.781349    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:31.932501 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:31.932465    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:32.118803 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:32.118774    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:32.375394 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:32.375223    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:32.782985 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:32.782890    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:34.449965 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:34.449922    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:36.749607 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:36.749573    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-6b3855/ipp-route-6b3855-kserve-f8c8cf4f7-4pnld"]
Oct 30 05:31:36.800077 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:36.800048    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:36.919939 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:36.917868    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:39.290869 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:39.290830    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:39.436311 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:39.436276    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:39.607200 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:39.607162    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:39.735296 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:39.735263    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:41.508476 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:41.508447    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:43.086265 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:43.086235    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:43.302517 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:43.302483    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:45.283652 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:45.283619    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:46.560480 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:46.560437    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:47.341419 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:47.341384    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:49.248759 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:49.248723    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:49.928385 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:49.928344    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:50.398633 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:50.398580    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:50.598384 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:50.598348    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:51.438805 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:51.438774    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:51.771839 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:51.771746    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-6b3855/ipp-route-6b3855-kserve-f8c8cf4f7-4pnld" podUID="0af0863a-6896-443d-be5a-bbb5a1e0edae" containerName="main" containerID="cri-o://5bcd9474c29a3f760f8ac4a5e3342b3972006392c4b63f0a4cc72e097dc08283" gracePeriod=45
Oct 30 05:31:52.073800 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:52.073773    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-6b3855/ipp-route-6b3855-kserve-f8c8cf4f7-4pnld"
Oct 30 05:31:52.130456 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:52.130422    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0af0863a-6896-443d-be5a-bbb5a1e0edae-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0af0863a-6896-443d-be5a-bbb5a1e0edae-tmp-dir\") pod \"0af0863a-6896-443d-be5a-bbb5a1e0edae\" (UID: \"0af0863a-6896-443d-be5a-bbb5a1e0edae\") "
Oct 30 05:31:52.130653 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:52.130509    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0af0863a-6896-443d-be5a-bbb5a1e0edae-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0af0863a-6896-443d-be5a-bbb5a1e0edae-model-cache\") pod \"0af0863a-6896-443d-be5a-bbb5a1e0edae\" (UID: \"0af0863a-6896-443d-be5a-bbb5a1e0edae\") "
Oct 30 05:31:52.130653 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:52.130580    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0af0863a-6896-443d-be5a-bbb5a1e0edae-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0af0863a-6896-443d-be5a-bbb5a1e0edae-dshm\") pod \"0af0863a-6896-443d-be5a-bbb5a1e0edae\" (UID: \"0af0863a-6896-443d-be5a-bbb5a1e0edae\") "
Oct 30 05:31:52.130653 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:52.130618    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0af0863a-6896-443d-be5a-bbb5a1e0edae-tls-certs\" (UniqueName: \"kubernetes.io/secret/0af0863a-6896-443d-be5a-bbb5a1e0edae-tls-certs\") pod \"0af0863a-6896-443d-be5a-bbb5a1e0edae\" (UID: \"0af0863a-6896-443d-be5a-bbb5a1e0edae\") "
Oct 30 05:31:52.130949 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:52.130656    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0af0863a-6896-443d-be5a-bbb5a1e0edae-home\" (UniqueName: \"kubernetes.io/empty-dir/0af0863a-6896-443d-be5a-bbb5a1e0edae-home\") pod \"0af0863a-6896-443d-be5a-bbb5a1e0edae\" (UID: \"0af0863a-6896-443d-be5a-bbb5a1e0edae\") "
Oct 30 05:31:52.130949 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:52.130723    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0af0863a-6896-443d-be5a-bbb5a1e0edae-kube-api-access-k5bhs\" (UniqueName: \"kubernetes.io/projected/0af0863a-6896-443d-be5a-bbb5a1e0edae-kube-api-access-k5bhs\") pod \"0af0863a-6896-443d-be5a-bbb5a1e0edae\" (UID: \"0af0863a-6896-443d-be5a-bbb5a1e0edae\") "
Oct 30 05:31:52.130949 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:52.130883    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0af0863a-6896-443d-be5a-bbb5a1e0edae-model-cache" pod "0af0863a-6896-443d-be5a-bbb5a1e0edae" (UID: "0af0863a-6896-443d-be5a-bbb5a1e0edae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:31:52.131156 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:52.131134    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0af0863a-6896-443d-be5a-bbb5a1e0edae-tmp-dir" pod "0af0863a-6896-443d-be5a-bbb5a1e0edae" (UID: "0af0863a-6896-443d-be5a-bbb5a1e0edae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:31:52.131500 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:52.131472    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0af0863a-6896-443d-be5a-bbb5a1e0edae-home" pod "0af0863a-6896-443d-be5a-bbb5a1e0edae" (UID: "0af0863a-6896-443d-be5a-bbb5a1e0edae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:31:52.131745 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:52.131726    5898 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0af0863a-6896-443d-be5a-bbb5a1e0edae-home\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:31:52.131830 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:52.131752    5898 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0af0863a-6896-443d-be5a-bbb5a1e0edae-tmp-dir\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:31:52.131830 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:52.131765    5898 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0af0863a-6896-443d-be5a-bbb5a1e0edae-model-cache\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:31:52.137350 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:52.137309    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0af0863a-6896-443d-be5a-bbb5a1e0edae-kube-api-access-k5bhs" pod "0af0863a-6896-443d-be5a-bbb5a1e0edae" (UID: "0af0863a-6896-443d-be5a-bbb5a1e0edae"). InnerVolumeSpecName "kube-api-access-k5bhs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:31:52.138261 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:52.138228    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0af0863a-6896-443d-be5a-bbb5a1e0edae-dshm" pod "0af0863a-6896-443d-be5a-bbb5a1e0edae" (UID: "0af0863a-6896-443d-be5a-bbb5a1e0edae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:31:52.142115 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:52.142090    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0af0863a-6896-443d-be5a-bbb5a1e0edae-tls-certs" pod "0af0863a-6896-443d-be5a-bbb5a1e0edae" (UID: "0af0863a-6896-443d-be5a-bbb5a1e0edae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:31:52.233430 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:52.233384    5898 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0af0863a-6896-443d-be5a-bbb5a1e0edae-dshm\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:31:52.233430 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:52.233430    5898 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0af0863a-6896-443d-be5a-bbb5a1e0edae-tls-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:31:52.233640 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:52.233443    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5bhs\" (UniqueName: \"kubernetes.io/projected/0af0863a-6896-443d-be5a-bbb5a1e0edae-kube-api-access-k5bhs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:31:52.595098 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:52.595062    5898 generic.go:356] "Generic (PLEG): container finished" podID="0af0863a-6896-443d-be5a-bbb5a1e0edae" containerID="5bcd9474c29a3f760f8ac4a5e3342b3972006392c4b63f0a4cc72e097dc08283" exitCode=0
Oct 30 05:31:52.595609 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:52.595112    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-6b3855/ipp-route-6b3855-kserve-f8c8cf4f7-4pnld"
Oct 30 05:31:52.595609 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:52.595139    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-6b3855/ipp-route-6b3855-kserve-f8c8cf4f7-4pnld" event={"ID":"0af0863a-6896-443d-be5a-bbb5a1e0edae","Type":"ContainerDied","Data":"5bcd9474c29a3f760f8ac4a5e3342b3972006392c4b63f0a4cc72e097dc08283"}
Oct 30 05:31:52.595609 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:52.595198    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-6b3855/ipp-route-6b3855-kserve-f8c8cf4f7-4pnld" event={"ID":"0af0863a-6896-443d-be5a-bbb5a1e0edae","Type":"ContainerDied","Data":"24a8b9aa783dcf86bc7154c111bc588fbe52e40f0dbd766a867342c6422acbee"}
Oct 30 05:31:52.595609 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:52.595222    5898 scope.go:122] "RemoveContainer" containerID="5bcd9474c29a3f760f8ac4a5e3342b3972006392c4b63f0a4cc72e097dc08283"
Oct 30 05:31:52.605833 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:52.605813    5898 scope.go:122] "RemoveContainer" containerID="5bcd9474c29a3f760f8ac4a5e3342b3972006392c4b63f0a4cc72e097dc08283"
Oct 30 05:31:52.606119 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:31:52.606098    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bcd9474c29a3f760f8ac4a5e3342b3972006392c4b63f0a4cc72e097dc08283\": container with ID starting with 5bcd9474c29a3f760f8ac4a5e3342b3972006392c4b63f0a4cc72e097dc08283 not found: ID does not exist" containerID="5bcd9474c29a3f760f8ac4a5e3342b3972006392c4b63f0a4cc72e097dc08283"
Oct 30 05:31:52.606174 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:52.606126    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bcd9474c29a3f760f8ac4a5e3342b3972006392c4b63f0a4cc72e097dc08283"} err="failed to get container status \"5bcd9474c29a3f760f8ac4a5e3342b3972006392c4b63f0a4cc72e097dc08283\": rpc error: code = NotFound desc = could not find container \"5bcd9474c29a3f760f8ac4a5e3342b3972006392c4b63f0a4cc72e097dc08283\": container with ID starting with 5bcd9474c29a3f760f8ac4a5e3342b3972006392c4b63f0a4cc72e097dc08283 not found: ID does not exist"
Oct 30 05:31:52.648216 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:52.648192    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-6b3855/ipp-route-6b3855-kserve-f8c8cf4f7-4pnld"]
Oct 30 05:31:52.659301 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:52.659274    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-6b3855/ipp-route-6b3855-kserve-f8c8cf4f7-4pnld"]
Oct 30 05:31:52.924309 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:52.924224    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0af0863a-6896-443d-be5a-bbb5a1e0edae" path="/var/lib/kubelet/pods/0af0863a-6896-443d-be5a-bbb5a1e0edae/volumes"
Oct 30 05:31:52.955484 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:52.955455    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:53.073373 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:53.073330    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:53.448055 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:53.448019    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:54.945483 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:54.945440    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:55.339150 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:55.339116    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:55.639601 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:55.639506    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:55.884222 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:55.884174    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:56.020896 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:56.020799    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:56.820327 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:31:56.820288    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:31:57.613388 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:57.613355    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:59.125716 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:59.125672    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:59.313122 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:59.313089    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:59.439709 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:59.439602    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:31:59.954820 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:31:59.954740    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:00.114962 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:00.114928    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:00.249081 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:00.243061    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:00.570166 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:00.570046    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:03.420967 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:03.420868    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:03.547414 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:03.547360    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:04.101524 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:04.101481    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:06.284806 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:06.284772    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:06.495104 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:06.495072    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b5b27e-6fc7474f79-lk2jv"]
Oct 30 05:32:06.495825 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:06.495796    5898 cpu_manager.go:414] "containerMap: removing container" podUID="0af0863a-6896-443d-be5a-bbb5a1e0edae" containerName="main"
Oct 30 05:32:06.495825 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:06.495821    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0af0863a-6896-443d-be5a-bbb5a1e0edae" containerName="main"
Oct 30 05:32:06.496023 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:06.495879    5898 cpu_manager.go:414] "containerMap: removing container" podUID="f05b65d9-9992-445b-af08-28759a4ef2e2" containerName="cleanup"
Oct 30 05:32:06.496023 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:06.495886    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f05b65d9-9992-445b-af08-28759a4ef2e2" containerName="cleanup"
Oct 30 05:32:06.496101 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:06.496048    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2b36ef9-dd65-43f2-88fe-f360f18ba0fc" containerName="cleanup"
Oct 30 05:32:06.496101 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:06.496062    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="f05b65d9-9992-445b-af08-28759a4ef2e2" containerName="cleanup"
Oct 30 05:32:06.496101 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:06.496076    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="0af0863a-6896-443d-be5a-bbb5a1e0edae" containerName="main"
Oct 30 05:32:06.499784 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:06.499765    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5b27e-6fc7474f79-lk2jv"
Oct 30 05:32:06.530813 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:06.530782    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b5b27e-dockercfg-8xxmp\""
Oct 30 05:32:06.537732 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:06.537666    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b5b27e-6fc7474f79-lk2jv"]
Oct 30 05:32:06.547931 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:06.547883    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkgn5\" (UniqueName: \"kubernetes.io/projected/4e8672db-7de8-4aee-8b3b-46ccdc97a211-kube-api-access-bkgn5\") pod \"payload-pre-processing-e2e-ipp-b-b5b27e-6fc7474f79-lk2jv\" (UID: \"4e8672db-7de8-4aee-8b3b-46ccdc97a211\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5b27e-6fc7474f79-lk2jv"
Oct 30 05:32:06.548089 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:06.548017    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e8672db-7de8-4aee-8b3b-46ccdc97a211-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b5b27e-6fc7474f79-lk2jv\" (UID: \"4e8672db-7de8-4aee-8b3b-46ccdc97a211\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5b27e-6fc7474f79-lk2jv"
Oct 30 05:32:06.591456 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:06.591424    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:06.616075 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:06.616023    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b5b27e\""
Oct 30 05:32:06.663923 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:06.663022    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkgn5\" (UniqueName: \"kubernetes.io/projected/4e8672db-7de8-4aee-8b3b-46ccdc97a211-kube-api-access-bkgn5\") pod \"payload-pre-processing-e2e-ipp-b-b5b27e-6fc7474f79-lk2jv\" (UID: \"4e8672db-7de8-4aee-8b3b-46ccdc97a211\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5b27e-6fc7474f79-lk2jv"
Oct 30 05:32:06.663923 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:06.663278    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e8672db-7de8-4aee-8b3b-46ccdc97a211-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b5b27e-6fc7474f79-lk2jv\" (UID: \"4e8672db-7de8-4aee-8b3b-46ccdc97a211\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5b27e-6fc7474f79-lk2jv"
Oct 30 05:32:06.681931 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:06.664877    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e8672db-7de8-4aee-8b3b-46ccdc97a211-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b5b27e-6fc7474f79-lk2jv\" (UID: \"4e8672db-7de8-4aee-8b3b-46ccdc97a211\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5b27e-6fc7474f79-lk2jv"
Oct 30 05:32:06.735012 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:06.734833    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:06.737256 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:06.737228    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkgn5\" (UniqueName: \"kubernetes.io/projected/4e8672db-7de8-4aee-8b3b-46ccdc97a211-kube-api-access-bkgn5\") pod \"payload-pre-processing-e2e-ipp-b-b5b27e-6fc7474f79-lk2jv\" (UID: \"4e8672db-7de8-4aee-8b3b-46ccdc97a211\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5b27e-6fc7474f79-lk2jv"
Oct 30 05:32:06.815272 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:06.815185    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5b27e-6fc7474f79-lk2jv"
Oct 30 05:32:07.368698 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:07.368667    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b5b27e-6fc7474f79-lk2jv"]
Oct 30 05:32:07.370178 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:32:07.370150    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e8672db_7de8_4aee_8b3b_46ccdc97a211.slice/crio-24d39e62b1d8963efc240796651879ba335e73592cb1675ebd36ca0368d221e0 WatchSource:0}: Error finding container 24d39e62b1d8963efc240796651879ba335e73592cb1675ebd36ca0368d221e0: Status 404 returned error can't find the container with id 24d39e62b1d8963efc240796651879ba335e73592cb1675ebd36ca0368d221e0
Oct 30 05:32:07.801129 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:07.801050    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5b27e-6fc7474f79-lk2jv" event={"ID":"4e8672db-7de8-4aee-8b3b-46ccdc97a211","Type":"ContainerStarted","Data":"728f6c799e487753ff890dad10f402f2e61eb26a10d5d0a11184a6447ec44fb6"}
Oct 30 05:32:07.801129 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:07.801109    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5b27e-6fc7474f79-lk2jv"
Oct 30 05:32:07.801325 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:07.801132    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5b27e-6fc7474f79-lk2jv" event={"ID":"4e8672db-7de8-4aee-8b3b-46ccdc97a211","Type":"ContainerStarted","Data":"24d39e62b1d8963efc240796651879ba335e73592cb1675ebd36ca0368d221e0"}
Oct 30 05:32:07.823413 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:07.823385    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:07.842020 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:07.841968    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5b27e-6fc7474f79-lk2jv" podStartSLOduration=1.841955716 podStartE2EDuration="1.841955716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:32:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:32:07.841832486 +0000 UTC m=+2857.065652387" watchObservedRunningTime="2026-10-30 05:32:07.841955716 +0000 UTC m=+2857.065775653"
Oct 30 05:32:07.985920 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:07.982803    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:08.201164 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:08.201119    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:08.813046 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:08.813017    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-097d90-jjmjs"]
Oct 30 05:32:08.817429 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:08.817413    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-097d90-jjmjs"
Oct 30 05:32:08.836678 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:08.836657    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-097d90-jjmjs"]
Oct 30 05:32:08.888234 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:08.888200    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba126756-27f8-4a1a-a87a-65915852ec6c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-097d90-jjmjs\" (UID: \"ba126756-27f8-4a1a-a87a-65915852ec6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-097d90-jjmjs"
Oct 30 05:32:08.888429 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:08.888244    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba126756-27f8-4a1a-a87a-65915852ec6c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-097d90-jjmjs\" (UID: \"ba126756-27f8-4a1a-a87a-65915852ec6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-097d90-jjmjs"
Oct 30 05:32:08.888429 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:08.888338    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9vwf\" (UniqueName: \"kubernetes.io/projected/ba126756-27f8-4a1a-a87a-65915852ec6c-kube-api-access-q9vwf\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-097d90-jjmjs\" (UID: \"ba126756-27f8-4a1a-a87a-65915852ec6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-097d90-jjmjs"
Oct 30 05:32:08.917865 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:08.917832    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:32:08.918103 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:08.918086    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\""
Oct 30 05:32:08.989691 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:08.989654    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba126756-27f8-4a1a-a87a-65915852ec6c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-097d90-jjmjs\" (UID: \"ba126756-27f8-4a1a-a87a-65915852ec6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-097d90-jjmjs"
Oct 30 05:32:08.989916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:08.989712    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba126756-27f8-4a1a-a87a-65915852ec6c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-097d90-jjmjs\" (UID: \"ba126756-27f8-4a1a-a87a-65915852ec6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-097d90-jjmjs"
Oct 30 05:32:08.989916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:08.989773    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9vwf\" (UniqueName: \"kubernetes.io/projected/ba126756-27f8-4a1a-a87a-65915852ec6c-kube-api-access-q9vwf\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-097d90-jjmjs\" (UID: \"ba126756-27f8-4a1a-a87a-65915852ec6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-097d90-jjmjs"
Oct 30 05:32:08.990319 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:08.990297    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba126756-27f8-4a1a-a87a-65915852ec6c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-097d90-jjmjs\" (UID: \"ba126756-27f8-4a1a-a87a-65915852ec6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-097d90-jjmjs"
Oct 30 05:32:08.992138 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:08.992111    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba126756-27f8-4a1a-a87a-65915852ec6c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-097d90-jjmjs\" (UID: \"ba126756-27f8-4a1a-a87a-65915852ec6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-097d90-jjmjs"
Oct 30 05:32:09.012779 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:09.012752    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9vwf\" (UniqueName: \"kubernetes.io/projected/ba126756-27f8-4a1a-a87a-65915852ec6c-kube-api-access-q9vwf\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-097d90-jjmjs\" (UID: \"ba126756-27f8-4a1a-a87a-65915852ec6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-097d90-jjmjs"
Oct 30 05:32:09.095469 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:09.095436    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:09.135926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:09.135873    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-097d90-jjmjs"
Oct 30 05:32:09.508440 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:32:09.508409    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba126756_27f8_4a1a_a87a_65915852ec6c.slice/crio-29d9ffae11cf805c8879854237813e221464eea073e79b82d20d93fdb38a2f29 WatchSource:0}: Error finding container 29d9ffae11cf805c8879854237813e221464eea073e79b82d20d93fdb38a2f29: Status 404 returned error can't find the container with id 29d9ffae11cf805c8879854237813e221464eea073e79b82d20d93fdb38a2f29
Oct 30 05:32:09.515485 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:09.515460    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-097d90-jjmjs"]
Oct 30 05:32:09.738244 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:09.738180    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:32:09.880045 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:09.880007    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-097d90-jjmjs" event={"ID":"ba126756-27f8-4a1a-a87a-65915852ec6c","Type":"ContainerStarted","Data":"29d9ffae11cf805c8879854237813e221464eea073e79b82d20d93fdb38a2f29"}
Oct 30 05:32:10.893571 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:10.893528    5898 generic.go:356] "Generic (PLEG): container finished" podID="ba126756-27f8-4a1a-a87a-65915852ec6c" containerID="fe5cb84299f6f28a53af1b34ffaeb2efa1ab7078dd6849ef371f0abfb25f9a1b" exitCode=0
Oct 30 05:32:10.895482 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:10.895457    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-097d90-jjmjs" event={"ID":"ba126756-27f8-4a1a-a87a-65915852ec6c","Type":"ContainerDied","Data":"fe5cb84299f6f28a53af1b34ffaeb2efa1ab7078dd6849ef371f0abfb25f9a1b"}
Oct 30 05:32:11.293271 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:11.293147    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:12.086201 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:12.086177    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-097d90-jjmjs"
Oct 30 05:32:12.169811 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:12.169733    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba126756-27f8-4a1a-a87a-65915852ec6c-kube-api-access-q9vwf\" (UniqueName: \"kubernetes.io/projected/ba126756-27f8-4a1a-a87a-65915852ec6c-kube-api-access-q9vwf\") pod \"ba126756-27f8-4a1a-a87a-65915852ec6c\" (UID: \"ba126756-27f8-4a1a-a87a-65915852ec6c\") "
Oct 30 05:32:12.169811 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:12.169770    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba126756-27f8-4a1a-a87a-65915852ec6c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba126756-27f8-4a1a-a87a-65915852ec6c-maas-api-openshift-ca\") pod \"ba126756-27f8-4a1a-a87a-65915852ec6c\" (UID: \"ba126756-27f8-4a1a-a87a-65915852ec6c\") "
Oct 30 05:32:12.170081 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:12.169861    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba126756-27f8-4a1a-a87a-65915852ec6c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba126756-27f8-4a1a-a87a-65915852ec6c-maas-api-xks-ca\") pod \"ba126756-27f8-4a1a-a87a-65915852ec6c\" (UID: \"ba126756-27f8-4a1a-a87a-65915852ec6c\") "
Oct 30 05:32:12.170411 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:12.170372    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba126756-27f8-4a1a-a87a-65915852ec6c-maas-api-openshift-ca" pod "ba126756-27f8-4a1a-a87a-65915852ec6c" (UID: "ba126756-27f8-4a1a-a87a-65915852ec6c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:32:12.171053 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:12.171025    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba126756-27f8-4a1a-a87a-65915852ec6c-maas-api-openshift-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:32:12.172260 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:12.172233    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba126756-27f8-4a1a-a87a-65915852ec6c-kube-api-access-q9vwf" pod "ba126756-27f8-4a1a-a87a-65915852ec6c" (UID: "ba126756-27f8-4a1a-a87a-65915852ec6c"). InnerVolumeSpecName "kube-api-access-q9vwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:32:12.172493 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:12.172469    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba126756-27f8-4a1a-a87a-65915852ec6c-maas-api-xks-ca" pod "ba126756-27f8-4a1a-a87a-65915852ec6c" (UID: "ba126756-27f8-4a1a-a87a-65915852ec6c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:32:12.273658 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:12.273599    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9vwf\" (UniqueName: \"kubernetes.io/projected/ba126756-27f8-4a1a-a87a-65915852ec6c-kube-api-access-q9vwf\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:32:12.273658 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:12.273636    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba126756-27f8-4a1a-a87a-65915852ec6c-maas-api-xks-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:32:13.030920 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:13.018821    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-097d90-jjmjs" event={"ID":"ba126756-27f8-4a1a-a87a-65915852ec6c","Type":"ContainerDied","Data":"29d9ffae11cf805c8879854237813e221464eea073e79b82d20d93fdb38a2f29"}
Oct 30 05:32:13.030920 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:13.022958    5898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29d9ffae11cf805c8879854237813e221464eea073e79b82d20d93fdb38a2f29"
Oct 30 05:32:13.030920 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:12.998015    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-097d90-jjmjs"
Oct 30 05:32:13.465335 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:13.465294    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:14.088250 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:14.088219    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:15.298105 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:15.298072    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:15.551757 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:15.551673    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:17.629528 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:17.629495    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:17.783642 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:17.783607    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:17.936652 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:17.936572    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:18.812200 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:18.812172    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5b27e-6fc7474f79-lk2jv"
Oct 30 05:32:19.629546 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:19.629498    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:19.805794 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:19.805761    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:20.008005 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:20.003935    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:20.631787 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:20.631755    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:20.791721 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:20.791681    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:22.008681 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:22.008649    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:22.106652 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:22.106609    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:23.671048 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:23.671012    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:23.779834 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:23.779801    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:23.984625 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:23.984550    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:25.917163 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:25.917131    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:26.820248 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:32:26.820207    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:32:27.737612 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:27.737576    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:28.891577 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:28.891537    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:29.012147 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:29.012117    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:29.282736 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:29.282653    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm"]
Oct 30 05:32:29.283213 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:29.283176    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm" podUID="1c26b723-62af-41b3-bf3a-36a48065a924" containerName="istio-proxy" containerID="cri-o://cf231194720d27a81df6e09d49ec938fcc546d22628fff97866e69bdd16498bc" gracePeriod=30
Oct 30 05:32:29.364997 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:29.364951    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:29.681144 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:29.660794    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:30.467447 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:30.467411    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:30.632188 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:30.632152    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:30.792837 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:30.792758    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-097d90-jjmjs"]
Oct 30 05:32:30.802948 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:30.802922    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-097d90-jjmjs"]
Oct 30 05:32:30.896070 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:30.896034    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba126756-27f8-4a1a-a87a-65915852ec6c" path="/var/lib/kubelet/pods/ba126756-27f8-4a1a-a87a-65915852ec6c/volumes"
Oct 30 05:32:31.052663 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:31.052580    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:31.286757 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:31.286728    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:34.055814 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.055777    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:34.614169 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.614145    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm"
Oct 30 05:32:34.688003 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.687965    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq"]
Oct 30 05:32:34.689077 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.689050    5898 cpu_manager.go:414] "containerMap: removing container" podUID="ba126756-27f8-4a1a-a87a-65915852ec6c" containerName="revoke-keys"
Oct 30 05:32:34.689077 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.689077    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba126756-27f8-4a1a-a87a-65915852ec6c" containerName="revoke-keys"
Oct 30 05:32:34.689232 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.689164    5898 cpu_manager.go:414] "containerMap: removing container" podUID="1c26b723-62af-41b3-bf3a-36a48065a924" containerName="istio-proxy"
Oct 30 05:32:34.689232 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.689172    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c26b723-62af-41b3-bf3a-36a48065a924" containerName="istio-proxy"
Oct 30 05:32:34.689347 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.689331    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba126756-27f8-4a1a-a87a-65915852ec6c" containerName="revoke-keys"
Oct 30 05:32:34.689385 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.689366    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c26b723-62af-41b3-bf3a-36a48065a924" containerName="istio-proxy"
Oct 30 05:32:34.693718 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.693696    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq"
Oct 30 05:32:34.718439 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.718407    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c26b723-62af-41b3-bf3a-36a48065a924-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c26b723-62af-41b3-bf3a-36a48065a924-istiod-ca-cert\") pod \"1c26b723-62af-41b3-bf3a-36a48065a924\" (UID: \"1c26b723-62af-41b3-bf3a-36a48065a924\") "
Oct 30 05:32:34.718652 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.718532    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c26b723-62af-41b3-bf3a-36a48065a924-istio-token\" (UniqueName: \"kubernetes.io/projected/1c26b723-62af-41b3-bf3a-36a48065a924-istio-token\") pod \"1c26b723-62af-41b3-bf3a-36a48065a924\" (UID: \"1c26b723-62af-41b3-bf3a-36a48065a924\") "
Oct 30 05:32:34.718652 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.718604    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c26b723-62af-41b3-bf3a-36a48065a924-kube-api-access-f6rcj\" (UniqueName: \"kubernetes.io/projected/1c26b723-62af-41b3-bf3a-36a48065a924-kube-api-access-f6rcj\") pod \"1c26b723-62af-41b3-bf3a-36a48065a924\" (UID: \"1c26b723-62af-41b3-bf3a-36a48065a924\") "
Oct 30 05:32:34.718755 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.718686    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c26b723-62af-41b3-bf3a-36a48065a924-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c26b723-62af-41b3-bf3a-36a48065a924-workload-certs\") pod \"1c26b723-62af-41b3-bf3a-36a48065a924\" (UID: \"1c26b723-62af-41b3-bf3a-36a48065a924\") "
Oct 30 05:32:34.718755 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.718717    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c26b723-62af-41b3-bf3a-36a48065a924-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c26b723-62af-41b3-bf3a-36a48065a924-istio-data\") pod \"1c26b723-62af-41b3-bf3a-36a48065a924\" (UID: \"1c26b723-62af-41b3-bf3a-36a48065a924\") "
Oct 30 05:32:34.718831 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.718813    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c26b723-62af-41b3-bf3a-36a48065a924-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c26b723-62af-41b3-bf3a-36a48065a924-credential-socket\") pod \"1c26b723-62af-41b3-bf3a-36a48065a924\" (UID: \"1c26b723-62af-41b3-bf3a-36a48065a924\") "
Oct 30 05:32:34.718958 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.718918    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c26b723-62af-41b3-bf3a-36a48065a924-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c26b723-62af-41b3-bf3a-36a48065a924-workload-socket\") pod \"1c26b723-62af-41b3-bf3a-36a48065a924\" (UID: \"1c26b723-62af-41b3-bf3a-36a48065a924\") "
Oct 30 05:32:34.719069 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.718975    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c26b723-62af-41b3-bf3a-36a48065a924-istiod-ca-cert" pod "1c26b723-62af-41b3-bf3a-36a48065a924" (UID: "1c26b723-62af-41b3-bf3a-36a48065a924"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:32:34.719069 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.718992    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c26b723-62af-41b3-bf3a-36a48065a924-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c26b723-62af-41b3-bf3a-36a48065a924-istio-envoy\") pod \"1c26b723-62af-41b3-bf3a-36a48065a924\" (UID: \"1c26b723-62af-41b3-bf3a-36a48065a924\") "
Oct 30 05:32:34.719069 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.719048    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1c26b723-62af-41b3-bf3a-36a48065a924-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c26b723-62af-41b3-bf3a-36a48065a924-istio-podinfo\") pod \"1c26b723-62af-41b3-bf3a-36a48065a924\" (UID: \"1c26b723-62af-41b3-bf3a-36a48065a924\") "
Oct 30 05:32:34.719909 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.719870    5898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c26b723-62af-41b3-bf3a-36a48065a924-istiod-ca-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:32:34.720003 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.719949    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c26b723-62af-41b3-bf3a-36a48065a924-workload-certs" pod "1c26b723-62af-41b3-bf3a-36a48065a924" (UID: "1c26b723-62af-41b3-bf3a-36a48065a924"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:32:34.738958 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.726519    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c26b723-62af-41b3-bf3a-36a48065a924-credential-socket" pod "1c26b723-62af-41b3-bf3a-36a48065a924" (UID: "1c26b723-62af-41b3-bf3a-36a48065a924"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:32:34.738958 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.726809    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c26b723-62af-41b3-bf3a-36a48065a924-istio-data" pod "1c26b723-62af-41b3-bf3a-36a48065a924" (UID: "1c26b723-62af-41b3-bf3a-36a48065a924"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:32:34.738958 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.727066    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c26b723-62af-41b3-bf3a-36a48065a924-workload-socket" pod "1c26b723-62af-41b3-bf3a-36a48065a924" (UID: "1c26b723-62af-41b3-bf3a-36a48065a924"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:32:34.738958 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.727106    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq"]
Oct 30 05:32:34.738958 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.727840    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45607d-5d4xh"]
Oct 30 05:32:34.738958 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.732209    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c26b723-62af-41b3-bf3a-36a48065a924-istio-envoy" pod "1c26b723-62af-41b3-bf3a-36a48065a924" (UID: "1c26b723-62af-41b3-bf3a-36a48065a924"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:32:34.738958 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.736290    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c26b723-62af-41b3-bf3a-36a48065a924-kube-api-access-f6rcj" pod "1c26b723-62af-41b3-bf3a-36a48065a924" (UID: "1c26b723-62af-41b3-bf3a-36a48065a924"). InnerVolumeSpecName "kube-api-access-f6rcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:32:34.738958 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.736550    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c26b723-62af-41b3-bf3a-36a48065a924-istio-token" pod "1c26b723-62af-41b3-bf3a-36a48065a924" (UID: "1c26b723-62af-41b3-bf3a-36a48065a924"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:32:34.738958 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.736868    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45607d-5d4xh"
Oct 30 05:32:34.738958 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.737498    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c26b723-62af-41b3-bf3a-36a48065a924-istio-podinfo" pod "1c26b723-62af-41b3-bf3a-36a48065a924" (UID: "1c26b723-62af-41b3-bf3a-36a48065a924"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:32:34.761920 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.761868    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45607d-5d4xh"]
Oct 30 05:32:34.762845 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.762133    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:32:34.762845 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.762224    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\""
Oct 30 05:32:34.821326 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.821278    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b386772-86df-4d1c-a5c4-fc27901eb9fd-workload-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq\" (UID: \"3b386772-86df-4d1c-a5c4-fc27901eb9fd\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq"
Oct 30 05:32:34.821536 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.821471    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mwss\" (UniqueName: \"kubernetes.io/projected/3b386772-86df-4d1c-a5c4-fc27901eb9fd-kube-api-access-5mwss\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq\" (UID: \"3b386772-86df-4d1c-a5c4-fc27901eb9fd\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq"
Oct 30 05:32:34.821536 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.821528    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e34f763e-383c-4de4-921b-416c969ec6f0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-45607d-5d4xh\" (UID: \"e34f763e-383c-4de4-921b-416c969ec6f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45607d-5d4xh"
Oct 30 05:32:34.821687 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.821596    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b386772-86df-4d1c-a5c4-fc27901eb9fd-istio-token\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq\" (UID: \"3b386772-86df-4d1c-a5c4-fc27901eb9fd\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq"
Oct 30 05:32:34.821687 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.821646    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e34f763e-383c-4de4-921b-416c969ec6f0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-45607d-5d4xh\" (UID: \"e34f763e-383c-4de4-921b-416c969ec6f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45607d-5d4xh"
Oct 30 05:32:34.821817 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.821685    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvbjs\" (UniqueName: \"kubernetes.io/projected/e34f763e-383c-4de4-921b-416c969ec6f0-kube-api-access-fvbjs\") pod \"maas-api-revoke-keys-e2e-ipp-b-45607d-5d4xh\" (UID: \"e34f763e-383c-4de4-921b-416c969ec6f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45607d-5d4xh"
Oct 30 05:32:34.821817 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.821716    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b386772-86df-4d1c-a5c4-fc27901eb9fd-credential-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq\" (UID: \"3b386772-86df-4d1c-a5c4-fc27901eb9fd\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq"
Oct 30 05:32:34.821993 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.821832    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b386772-86df-4d1c-a5c4-fc27901eb9fd-istio-data\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq\" (UID: \"3b386772-86df-4d1c-a5c4-fc27901eb9fd\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq"
Oct 30 05:32:34.822065 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.822021    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b386772-86df-4d1c-a5c4-fc27901eb9fd-istiod-ca-cert\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq\" (UID: \"3b386772-86df-4d1c-a5c4-fc27901eb9fd\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq"
Oct 30 05:32:34.822129 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.822100    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b386772-86df-4d1c-a5c4-fc27901eb9fd-istio-envoy\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq\" (UID: \"3b386772-86df-4d1c-a5c4-fc27901eb9fd\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq"
Oct 30 05:32:34.822188 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.822176    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b386772-86df-4d1c-a5c4-fc27901eb9fd-workload-certs\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq\" (UID: \"3b386772-86df-4d1c-a5c4-fc27901eb9fd\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq"
Oct 30 05:32:34.822233 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.822193    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b386772-86df-4d1c-a5c4-fc27901eb9fd-istio-podinfo\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq\" (UID: \"3b386772-86df-4d1c-a5c4-fc27901eb9fd\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq"
Oct 30 05:32:34.822354 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.822337    5898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c26b723-62af-41b3-bf3a-36a48065a924-istio-envoy\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:32:34.822417 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.822363    5898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c26b723-62af-41b3-bf3a-36a48065a924-istio-podinfo\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:32:34.822417 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.822374    5898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c26b723-62af-41b3-bf3a-36a48065a924-istio-token\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:32:34.822417 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.822383    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6rcj\" (UniqueName: \"kubernetes.io/projected/1c26b723-62af-41b3-bf3a-36a48065a924-kube-api-access-f6rcj\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:32:34.822417 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.822391    5898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c26b723-62af-41b3-bf3a-36a48065a924-workload-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:32:34.822417 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.822400    5898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c26b723-62af-41b3-bf3a-36a48065a924-istio-data\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:32:34.822417 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.822408    5898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c26b723-62af-41b3-bf3a-36a48065a924-credential-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:32:34.822417 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.822417    5898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c26b723-62af-41b3-bf3a-36a48065a924-workload-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:32:34.923953 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.923920    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b386772-86df-4d1c-a5c4-fc27901eb9fd-workload-certs\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq\" (UID: \"3b386772-86df-4d1c-a5c4-fc27901eb9fd\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq"
Oct 30 05:32:34.924099 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.923966    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b386772-86df-4d1c-a5c4-fc27901eb9fd-istio-podinfo\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq\" (UID: \"3b386772-86df-4d1c-a5c4-fc27901eb9fd\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq"
Oct 30 05:32:34.924099 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.924046    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b386772-86df-4d1c-a5c4-fc27901eb9fd-workload-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq\" (UID: \"3b386772-86df-4d1c-a5c4-fc27901eb9fd\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq"
Oct 30 05:32:34.924099 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.924071    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mwss\" (UniqueName: \"kubernetes.io/projected/3b386772-86df-4d1c-a5c4-fc27901eb9fd-kube-api-access-5mwss\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq\" (UID: \"3b386772-86df-4d1c-a5c4-fc27901eb9fd\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq"
Oct 30 05:32:34.924252 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.924110    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e34f763e-383c-4de4-921b-416c969ec6f0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-45607d-5d4xh\" (UID: \"e34f763e-383c-4de4-921b-416c969ec6f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45607d-5d4xh"
Oct 30 05:32:34.924252 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.924163    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b386772-86df-4d1c-a5c4-fc27901eb9fd-istio-token\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq\" (UID: \"3b386772-86df-4d1c-a5c4-fc27901eb9fd\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq"
Oct 30 05:32:34.924252 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.924227    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e34f763e-383c-4de4-921b-416c969ec6f0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-45607d-5d4xh\" (UID: \"e34f763e-383c-4de4-921b-416c969ec6f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45607d-5d4xh"
Oct 30 05:32:34.924424 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.924259    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b386772-86df-4d1c-a5c4-fc27901eb9fd-workload-certs\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq\" (UID: \"3b386772-86df-4d1c-a5c4-fc27901eb9fd\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq"
Oct 30 05:32:34.924424 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.924280    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvbjs\" (UniqueName: \"kubernetes.io/projected/e34f763e-383c-4de4-921b-416c969ec6f0-kube-api-access-fvbjs\") pod \"maas-api-revoke-keys-e2e-ipp-b-45607d-5d4xh\" (UID: \"e34f763e-383c-4de4-921b-416c969ec6f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45607d-5d4xh"
Oct 30 05:32:34.924424 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.924323    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b386772-86df-4d1c-a5c4-fc27901eb9fd-credential-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq\" (UID: \"3b386772-86df-4d1c-a5c4-fc27901eb9fd\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq"
Oct 30 05:32:34.924424 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.924355    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b386772-86df-4d1c-a5c4-fc27901eb9fd-istio-data\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq\" (UID: \"3b386772-86df-4d1c-a5c4-fc27901eb9fd\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq"
Oct 30 05:32:34.924424 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.924388    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b386772-86df-4d1c-a5c4-fc27901eb9fd-workload-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq\" (UID: \"3b386772-86df-4d1c-a5c4-fc27901eb9fd\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq"
Oct 30 05:32:34.924715 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.924529    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b386772-86df-4d1c-a5c4-fc27901eb9fd-istiod-ca-cert\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq\" (UID: \"3b386772-86df-4d1c-a5c4-fc27901eb9fd\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq"
Oct 30 05:32:34.924715 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.924579    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b386772-86df-4d1c-a5c4-fc27901eb9fd-istio-envoy\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq\" (UID: \"3b386772-86df-4d1c-a5c4-fc27901eb9fd\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq"
Oct 30 05:32:34.924715 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.924702    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b386772-86df-4d1c-a5c4-fc27901eb9fd-credential-socket\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq\" (UID: \"3b386772-86df-4d1c-a5c4-fc27901eb9fd\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq"
Oct 30 05:32:34.924915 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.924723    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b386772-86df-4d1c-a5c4-fc27901eb9fd-istio-data\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq\" (UID: \"3b386772-86df-4d1c-a5c4-fc27901eb9fd\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq"
Oct 30 05:32:34.925018 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.924996    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e34f763e-383c-4de4-921b-416c969ec6f0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-45607d-5d4xh\" (UID: \"e34f763e-383c-4de4-921b-416c969ec6f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45607d-5d4xh"
Oct 30 05:32:34.925396 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.925372    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b386772-86df-4d1c-a5c4-fc27901eb9fd-istiod-ca-cert\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq\" (UID: \"3b386772-86df-4d1c-a5c4-fc27901eb9fd\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq"
Oct 30 05:32:34.926670 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.926652    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e34f763e-383c-4de4-921b-416c969ec6f0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-45607d-5d4xh\" (UID: \"e34f763e-383c-4de4-921b-416c969ec6f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45607d-5d4xh"
Oct 30 05:32:34.926896 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.926881    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b386772-86df-4d1c-a5c4-fc27901eb9fd-istio-envoy\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq\" (UID: \"3b386772-86df-4d1c-a5c4-fc27901eb9fd\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq"
Oct 30 05:32:34.927024 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.927008    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b386772-86df-4d1c-a5c4-fc27901eb9fd-istio-podinfo\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq\" (UID: \"3b386772-86df-4d1c-a5c4-fc27901eb9fd\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq"
Oct 30 05:32:34.942344 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.942325    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mwss\" (UniqueName: \"kubernetes.io/projected/3b386772-86df-4d1c-a5c4-fc27901eb9fd-kube-api-access-5mwss\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq\" (UID: \"3b386772-86df-4d1c-a5c4-fc27901eb9fd\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq"
Oct 30 05:32:34.942533 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.942517    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b386772-86df-4d1c-a5c4-fc27901eb9fd-istio-token\") pod \"e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq\" (UID: \"3b386772-86df-4d1c-a5c4-fc27901eb9fd\") " pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq"
Oct 30 05:32:34.945358 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:34.945341    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvbjs\" (UniqueName: \"kubernetes.io/projected/e34f763e-383c-4de4-921b-416c969ec6f0-kube-api-access-fvbjs\") pod \"maas-api-revoke-keys-e2e-ipp-b-45607d-5d4xh\" (UID: \"e34f763e-383c-4de4-921b-416c969ec6f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45607d-5d4xh"
Oct 30 05:32:35.010038 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:35.010011    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq"
Oct 30 05:32:35.100399 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:35.100357    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45607d-5d4xh"
Oct 30 05:32:35.227408 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:35.227372    5898 generic.go:356] "Generic (PLEG): container finished" podID="1c26b723-62af-41b3-bf3a-36a48065a924" containerID="cf231194720d27a81df6e09d49ec938fcc546d22628fff97866e69bdd16498bc" exitCode=0
Oct 30 05:32:35.227576 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:35.227442    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm"
Oct 30 05:32:35.227576 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:35.227468    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm" event={"ID":"1c26b723-62af-41b3-bf3a-36a48065a924","Type":"ContainerDied","Data":"cf231194720d27a81df6e09d49ec938fcc546d22628fff97866e69bdd16498bc"}
Oct 30 05:32:35.227576 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:35.227537    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm" event={"ID":"1c26b723-62af-41b3-bf3a-36a48065a924","Type":"ContainerDied","Data":"e581fd588f825cdb4c70355a11d45c6ee6a59e973ab7c6294225e83b2654809e"}
Oct 30 05:32:35.227576 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:35.227575    5898 scope.go:122] "RemoveContainer" containerID="cf231194720d27a81df6e09d49ec938fcc546d22628fff97866e69bdd16498bc"
Oct 30 05:32:35.245178 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:35.245158    5898 scope.go:122] "RemoveContainer" containerID="cf231194720d27a81df6e09d49ec938fcc546d22628fff97866e69bdd16498bc"
Oct 30 05:32:35.245553 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:32:35.245525    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf231194720d27a81df6e09d49ec938fcc546d22628fff97866e69bdd16498bc\": container with ID starting with cf231194720d27a81df6e09d49ec938fcc546d22628fff97866e69bdd16498bc not found: ID does not exist" containerID="cf231194720d27a81df6e09d49ec938fcc546d22628fff97866e69bdd16498bc"
Oct 30 05:32:35.245699 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:35.245561    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf231194720d27a81df6e09d49ec938fcc546d22628fff97866e69bdd16498bc"} err="failed to get container status \"cf231194720d27a81df6e09d49ec938fcc546d22628fff97866e69bdd16498bc\": rpc error: code = NotFound desc = could not find container \"cf231194720d27a81df6e09d49ec938fcc546d22628fff97866e69bdd16498bc\": container with ID starting with cf231194720d27a81df6e09d49ec938fcc546d22628fff97866e69bdd16498bc not found: ID does not exist"
Oct 30 05:32:35.285312 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:35.285282    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm"]
Oct 30 05:32:35.296422 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:35.296394    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-6b3855-openshift-default-5f877686d5-gbwjm"]
Oct 30 05:32:35.342495 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:35.342467    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq"]
Oct 30 05:32:35.344558 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:32:35.344536    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b386772_86df_4d1c_a5c4_fc27901eb9fd.slice/crio-6791d6fabae85ee6302a7c3c80b4bcc48d552e82148800f90bdee84d5d1077fd WatchSource:0}: Error finding container 6791d6fabae85ee6302a7c3c80b4bcc48d552e82148800f90bdee84d5d1077fd: Status 404 returned error can't find the container with id 6791d6fabae85ee6302a7c3c80b4bcc48d552e82148800f90bdee84d5d1077fd
Oct 30 05:32:35.348135 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:35.348102    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:32:35.348224 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:35.348184    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:32:35.348224 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:35.348213    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:32:35.418223 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:35.418171    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45607d-5d4xh"]
Oct 30 05:32:35.422438 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:32:35.422408    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode34f763e_383c_4de4_921b_416c969ec6f0.slice/crio-c608e35038d58f87f81fe59388c19a18d6b4a837f67ee0bf9ac8ee6a9f3d43c0 WatchSource:0}: Error finding container c608e35038d58f87f81fe59388c19a18d6b4a837f67ee0bf9ac8ee6a9f3d43c0: Status 404 returned error can't find the container with id c608e35038d58f87f81fe59388c19a18d6b4a837f67ee0bf9ac8ee6a9f3d43c0
Oct 30 05:32:36.236777 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:36.236733    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45607d-5d4xh" event={"ID":"e34f763e-383c-4de4-921b-416c969ec6f0","Type":"ContainerStarted","Data":"2aafcc2a4818acbb0e27c615686c629d50d01edc3a9035638051e885ab4e7e82"}
Oct 30 05:32:36.237426 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:36.236804    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45607d-5d4xh" event={"ID":"e34f763e-383c-4de4-921b-416c969ec6f0","Type":"ContainerStarted","Data":"c608e35038d58f87f81fe59388c19a18d6b4a837f67ee0bf9ac8ee6a9f3d43c0"}
Oct 30 05:32:36.239735 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:36.239708    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq" event={"ID":"3b386772-86df-4d1c-a5c4-fc27901eb9fd","Type":"ContainerStarted","Data":"4d3d1302c993c1ba8acdb8a90fd594be0ea096410e51f772f396b3dccd3e69e8"}
Oct 30 05:32:36.239877 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:36.239758    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq" event={"ID":"3b386772-86df-4d1c-a5c4-fc27901eb9fd","Type":"ContainerStarted","Data":"6791d6fabae85ee6302a7c3c80b4bcc48d552e82148800f90bdee84d5d1077fd"}
Oct 30 05:32:36.362765 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:36.362703    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45607d-5d4xh" podStartSLOduration=2.362673421 podStartE2EDuration="2.362673421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:32:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:32:36.297300257 +0000 UTC m=+2885.521120161" watchObservedRunningTime="2026-10-30 05:32:36.362673421 +0000 UTC m=+2885.586493322"
Oct 30 05:32:36.366668 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:36.366628    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq" podStartSLOduration=132.366615958 podStartE2EDuration="2m12.366615958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:32:36.361577237 +0000 UTC m=+2885.585397158" watchObservedRunningTime="2026-10-30 05:32:36.366615958 +0000 UTC m=+2885.590435859"
Oct 30 05:32:36.469012 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:36.468619    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:36.909838 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:36.909446    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c26b723-62af-41b3-bf3a-36a48065a924" path="/var/lib/kubelet/pods/1c26b723-62af-41b3-bf3a-36a48065a924/volumes"
Oct 30 05:32:37.010713 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:37.010681    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq"
Oct 30 05:32:37.012200 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:37.012178    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body=
Oct 30 05:32:37.012267 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:37.012224    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq" podUID="3b386772-86df-4d1c-a5c4-fc27901eb9fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused"
Oct 30 05:32:37.249510 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:37.249424    5898 generic.go:356] "Generic (PLEG): container finished" podID="e34f763e-383c-4de4-921b-416c969ec6f0" containerID="2aafcc2a4818acbb0e27c615686c629d50d01edc3a9035638051e885ab4e7e82" exitCode=0
Oct 30 05:32:37.249876 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:37.249500    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45607d-5d4xh" event={"ID":"e34f763e-383c-4de4-921b-416c969ec6f0","Type":"ContainerDied","Data":"2aafcc2a4818acbb0e27c615686c629d50d01edc3a9035638051e885ab4e7e82"}
Oct 30 05:32:37.920456 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:37.920424    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:38.029207 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:38.029167    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body=
Oct 30 05:32:38.029504 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:38.029470    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq" podUID="3b386772-86df-4d1c-a5c4-fc27901eb9fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused"
Oct 30 05:32:38.107824 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:38.107787    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:38.461017 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:38.448500    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:38.645417 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:38.645334    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:39.011242 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:39.011212    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body=
Oct 30 05:32:39.011430 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:39.011264    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq" podUID="3b386772-86df-4d1c-a5c4-fc27901eb9fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused"
Oct 30 05:32:39.021087 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:39.021057    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45607d-5d4xh"
Oct 30 05:32:39.032274 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:39.032245    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e34f763e-383c-4de4-921b-416c969ec6f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e34f763e-383c-4de4-921b-416c969ec6f0-maas-api-xks-ca\") pod \"e34f763e-383c-4de4-921b-416c969ec6f0\" (UID: \"e34f763e-383c-4de4-921b-416c969ec6f0\") "
Oct 30 05:32:39.032515 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:39.032494    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e34f763e-383c-4de4-921b-416c969ec6f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e34f763e-383c-4de4-921b-416c969ec6f0-maas-api-openshift-ca\") pod \"e34f763e-383c-4de4-921b-416c969ec6f0\" (UID: \"e34f763e-383c-4de4-921b-416c969ec6f0\") "
Oct 30 05:32:39.032777 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:39.032694    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e34f763e-383c-4de4-921b-416c969ec6f0-kube-api-access-fvbjs\" (UniqueName: \"kubernetes.io/projected/e34f763e-383c-4de4-921b-416c969ec6f0-kube-api-access-fvbjs\") pod \"e34f763e-383c-4de4-921b-416c969ec6f0\" (UID: \"e34f763e-383c-4de4-921b-416c969ec6f0\") "
Oct 30 05:32:39.033755 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:39.033731    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e34f763e-383c-4de4-921b-416c969ec6f0-maas-api-openshift-ca" pod "e34f763e-383c-4de4-921b-416c969ec6f0" (UID: "e34f763e-383c-4de4-921b-416c969ec6f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:32:39.035405 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:39.035377    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e34f763e-383c-4de4-921b-416c969ec6f0-maas-api-xks-ca" pod "e34f763e-383c-4de4-921b-416c969ec6f0" (UID: "e34f763e-383c-4de4-921b-416c969ec6f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:32:39.035992 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:39.035967    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e34f763e-383c-4de4-921b-416c969ec6f0-kube-api-access-fvbjs" pod "e34f763e-383c-4de4-921b-416c969ec6f0" (UID: "e34f763e-383c-4de4-921b-416c969ec6f0"). InnerVolumeSpecName "kube-api-access-fvbjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:32:39.139645 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:39.134951    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvbjs\" (UniqueName: \"kubernetes.io/projected/e34f763e-383c-4de4-921b-416c969ec6f0-kube-api-access-fvbjs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:32:39.139645 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:39.138576    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e34f763e-383c-4de4-921b-416c969ec6f0-maas-api-xks-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:32:39.139645 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:39.138602    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e34f763e-383c-4de4-921b-416c969ec6f0-maas-api-openshift-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:32:39.325484 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:39.325459    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45607d-5d4xh"
Oct 30 05:32:39.325484 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:39.325472    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45607d-5d4xh" event={"ID":"e34f763e-383c-4de4-921b-416c969ec6f0","Type":"ContainerDied","Data":"c608e35038d58f87f81fe59388c19a18d6b4a837f67ee0bf9ac8ee6a9f3d43c0"}
Oct 30 05:32:39.325694 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:39.325516    5898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c608e35038d58f87f81fe59388c19a18d6b4a837f67ee0bf9ac8ee6a9f3d43c0"
Oct 30 05:32:39.606643 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:39.606492    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:40.010959 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:40.010869    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body=
Oct 30 05:32:40.010959 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:40.010938    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq" podUID="3b386772-86df-4d1c-a5c4-fc27901eb9fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused"
Oct 30 05:32:41.010514 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:41.010484    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body=
Oct 30 05:32:41.010929 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:41.010536    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq" podUID="3b386772-86df-4d1c-a5c4-fc27901eb9fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused"
Oct 30 05:32:41.980429 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:41.980392    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:42.010754 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:42.010721    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body=
Oct 30 05:32:42.019482 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:42.010779    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq" podUID="3b386772-86df-4d1c-a5c4-fc27901eb9fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused"
Oct 30 05:32:43.016651 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:43.016622    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq"
Oct 30 05:32:43.017110 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:43.017093    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq"
Oct 30 05:32:43.017701 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:43.017683    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq"
Oct 30 05:32:43.219718 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:43.219684    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:43.518518 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:43.518482    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:43.780489 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:43.780333    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:44.962648 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:44.962618    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:46.920568 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:46.920523    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:48.939217 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:48.939157    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:49.371048 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:49.371014    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e15c99-79bc65cccb-gwzkf"]
Oct 30 05:32:49.529771 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:49.529735    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:49.650757 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:49.650658    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:51.101130 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:51.101093    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:52.980028 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:52.970259    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:54.371860 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:54.371818    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e15c99-79bc65cccb-gwzkf" podUID="ef150e5d-807c-4d9b-9dd0-a3e4953e1e21" containerName="payload-processing" containerID="cri-o://0845d47eb8aafcb7781ef2d5d94330ba9841b4e544f086e0d8a4abd79688283e" gracePeriod=25
Oct 30 05:32:54.670162 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:54.670133    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e15c99-79bc65cccb-gwzkf"
Oct 30 05:32:54.730306 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:54.730278    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-02cffa-7875b685ff-pchf5"]
Oct 30 05:32:54.731030 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:54.731015    5898 cpu_manager.go:414] "containerMap: removing container" podUID="e34f763e-383c-4de4-921b-416c969ec6f0" containerName="revoke-keys"
Oct 30 05:32:54.731085 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:54.731031    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e34f763e-383c-4de4-921b-416c969ec6f0" containerName="revoke-keys"
Oct 30 05:32:54.731085 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:54.731054    5898 cpu_manager.go:414] "containerMap: removing container" podUID="ef150e5d-807c-4d9b-9dd0-a3e4953e1e21" containerName="payload-processing"
Oct 30 05:32:54.731085 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:54.731059    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef150e5d-807c-4d9b-9dd0-a3e4953e1e21" containerName="payload-processing"
Oct 30 05:32:54.731189 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:54.731169    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef150e5d-807c-4d9b-9dd0-a3e4953e1e21" containerName="payload-processing"
Oct 30 05:32:54.731227 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:54.731188    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="e34f763e-383c-4de4-921b-416c969ec6f0" containerName="revoke-keys"
Oct 30 05:32:54.734647 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:54.734629    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02cffa-7875b685ff-pchf5"
Oct 30 05:32:54.749987 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:54.749965    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-02cffa-dockercfg-7sxhb\""
Oct 30 05:32:54.751414 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:54.751393    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-02cffa\""
Oct 30 05:32:54.756220 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:54.756197    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-02cffa-7875b685ff-pchf5"]
Oct 30 05:32:54.758555 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:54.758527    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef150e5d-807c-4d9b-9dd0-a3e4953e1e21-kube-api-access-4vws6\" (UniqueName: \"kubernetes.io/projected/ef150e5d-807c-4d9b-9dd0-a3e4953e1e21-kube-api-access-4vws6\") pod \"ef150e5d-807c-4d9b-9dd0-a3e4953e1e21\" (UID: \"ef150e5d-807c-4d9b-9dd0-a3e4953e1e21\") "
Oct 30 05:32:54.758756 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:54.758728    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef150e5d-807c-4d9b-9dd0-a3e4953e1e21-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef150e5d-807c-4d9b-9dd0-a3e4953e1e21-plugins-config-volume\") pod \"ef150e5d-807c-4d9b-9dd0-a3e4953e1e21\" (UID: \"ef150e5d-807c-4d9b-9dd0-a3e4953e1e21\") "
Oct 30 05:32:54.759683 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:54.759349    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h798n\" (UniqueName: \"kubernetes.io/projected/4f19c225-09ec-43b9-af12-7c05b7546d72-kube-api-access-h798n\") pod \"payload-pre-processing-e2e-ipp-a-02cffa-7875b685ff-pchf5\" (UID: \"4f19c225-09ec-43b9-af12-7c05b7546d72\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02cffa-7875b685ff-pchf5"
Oct 30 05:32:54.759683 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:54.759481    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f19c225-09ec-43b9-af12-7c05b7546d72-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-02cffa-7875b685ff-pchf5\" (UID: \"4f19c225-09ec-43b9-af12-7c05b7546d72\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02cffa-7875b685ff-pchf5"
Oct 30 05:32:54.759979 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:54.759945    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef150e5d-807c-4d9b-9dd0-a3e4953e1e21-plugins-config-volume" pod "ef150e5d-807c-4d9b-9dd0-a3e4953e1e21" (UID: "ef150e5d-807c-4d9b-9dd0-a3e4953e1e21"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:32:54.771301 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:54.771273    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef150e5d-807c-4d9b-9dd0-a3e4953e1e21-kube-api-access-4vws6" pod "ef150e5d-807c-4d9b-9dd0-a3e4953e1e21" (UID: "ef150e5d-807c-4d9b-9dd0-a3e4953e1e21"). InnerVolumeSpecName "kube-api-access-4vws6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:32:54.861072 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:54.861040    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f19c225-09ec-43b9-af12-7c05b7546d72-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-02cffa-7875b685ff-pchf5\" (UID: \"4f19c225-09ec-43b9-af12-7c05b7546d72\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02cffa-7875b685ff-pchf5"
Oct 30 05:32:54.861292 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:54.861279    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h798n\" (UniqueName: \"kubernetes.io/projected/4f19c225-09ec-43b9-af12-7c05b7546d72-kube-api-access-h798n\") pod \"payload-pre-processing-e2e-ipp-a-02cffa-7875b685ff-pchf5\" (UID: \"4f19c225-09ec-43b9-af12-7c05b7546d72\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02cffa-7875b685ff-pchf5"
Oct 30 05:32:54.861359 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:54.861339    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vws6\" (UniqueName: \"kubernetes.io/projected/ef150e5d-807c-4d9b-9dd0-a3e4953e1e21-kube-api-access-4vws6\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:32:54.861359 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:54.861354    5898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef150e5d-807c-4d9b-9dd0-a3e4953e1e21-plugins-config-volume\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:32:54.861770 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:54.861751    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f19c225-09ec-43b9-af12-7c05b7546d72-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-02cffa-7875b685ff-pchf5\" (UID: \"4f19c225-09ec-43b9-af12-7c05b7546d72\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02cffa-7875b685ff-pchf5"
Oct 30 05:32:54.880370 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:54.880342    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h798n\" (UniqueName: \"kubernetes.io/projected/4f19c225-09ec-43b9-af12-7c05b7546d72-kube-api-access-h798n\") pod \"payload-pre-processing-e2e-ipp-a-02cffa-7875b685ff-pchf5\" (UID: \"4f19c225-09ec-43b9-af12-7c05b7546d72\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02cffa-7875b685ff-pchf5"
Oct 30 05:32:54.931369 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:54.931334    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:55.052934 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:55.052883    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02cffa-7875b685ff-pchf5"
Oct 30 05:32:55.372460 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:55.372431    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-02cffa-7875b685ff-pchf5"]
Oct 30 05:32:55.374036 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:32:55.374010    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f19c225_09ec_43b9_af12_7c05b7546d72.slice/crio-661ad3ee3c4c3d0661cc08dfcb3d002e65e8bf6fc3a8fd2475be933c8f4fd3d4 WatchSource:0}: Error finding container 661ad3ee3c4c3d0661cc08dfcb3d002e65e8bf6fc3a8fd2475be933c8f4fd3d4: Status 404 returned error can't find the container with id 661ad3ee3c4c3d0661cc08dfcb3d002e65e8bf6fc3a8fd2475be933c8f4fd3d4
Oct 30 05:32:55.376031 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:55.376013    5898 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 05:32:55.468283 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:55.468246    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02cffa-7875b685ff-pchf5" event={"ID":"4f19c225-09ec-43b9-af12-7c05b7546d72","Type":"ContainerStarted","Data":"661ad3ee3c4c3d0661cc08dfcb3d002e65e8bf6fc3a8fd2475be933c8f4fd3d4"}
Oct 30 05:32:55.469678 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:55.469651    5898 generic.go:356] "Generic (PLEG): container finished" podID="ef150e5d-807c-4d9b-9dd0-a3e4953e1e21" containerID="0845d47eb8aafcb7781ef2d5d94330ba9841b4e544f086e0d8a4abd79688283e" exitCode=0
Oct 30 05:32:55.469810 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:55.469698    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e15c99-79bc65cccb-gwzkf"
Oct 30 05:32:55.469810 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:55.469740    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e15c99-79bc65cccb-gwzkf" event={"ID":"ef150e5d-807c-4d9b-9dd0-a3e4953e1e21","Type":"ContainerDied","Data":"0845d47eb8aafcb7781ef2d5d94330ba9841b4e544f086e0d8a4abd79688283e"}
Oct 30 05:32:55.469810 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:55.469793    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e15c99-79bc65cccb-gwzkf" event={"ID":"ef150e5d-807c-4d9b-9dd0-a3e4953e1e21","Type":"ContainerDied","Data":"54097021c2cff13df0d974fafadbe836e830774a7ff4148b580e8fe8692ad989"}
Oct 30 05:32:55.470026 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:55.469823    5898 scope.go:122] "RemoveContainer" containerID="0845d47eb8aafcb7781ef2d5d94330ba9841b4e544f086e0d8a4abd79688283e"
Oct 30 05:32:55.480936 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:55.480884    5898 scope.go:122] "RemoveContainer" containerID="0845d47eb8aafcb7781ef2d5d94330ba9841b4e544f086e0d8a4abd79688283e"
Oct 30 05:32:55.481187 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:32:55.481159    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0845d47eb8aafcb7781ef2d5d94330ba9841b4e544f086e0d8a4abd79688283e\": container with ID starting with 0845d47eb8aafcb7781ef2d5d94330ba9841b4e544f086e0d8a4abd79688283e not found: ID does not exist" containerID="0845d47eb8aafcb7781ef2d5d94330ba9841b4e544f086e0d8a4abd79688283e"
Oct 30 05:32:55.481238 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:55.481189    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0845d47eb8aafcb7781ef2d5d94330ba9841b4e544f086e0d8a4abd79688283e"} err="failed to get container status \"0845d47eb8aafcb7781ef2d5d94330ba9841b4e544f086e0d8a4abd79688283e\": rpc error: code = NotFound desc = could not find container \"0845d47eb8aafcb7781ef2d5d94330ba9841b4e544f086e0d8a4abd79688283e\": container with ID starting with 0845d47eb8aafcb7781ef2d5d94330ba9841b4e544f086e0d8a4abd79688283e not found: ID does not exist"
Oct 30 05:32:55.526244 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:55.526221    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e15c99-79bc65cccb-gwzkf"]
Oct 30 05:32:55.532535 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:55.532515    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e15c99-79bc65cccb-gwzkf"]
Oct 30 05:32:56.541799 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:56.541723    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:56.971347 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:56.971314    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef150e5d-807c-4d9b-9dd0-a3e4953e1e21" path="/var/lib/kubelet/pods/ef150e5d-807c-4d9b-9dd0-a3e4953e1e21/volumes"
Oct 30 05:32:57.047933 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:32:57.028093    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:32:57.438412 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:57.438375    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:57.510446 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:57.510414    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02cffa-7875b685ff-pchf5" event={"ID":"4f19c225-09ec-43b9-af12-7c05b7546d72","Type":"ContainerStarted","Data":"5e6eb7c72854600ad6ef9e8dfa0abf57bd140a3ed92a833063a2cd0ae6cd29a3"}
Oct 30 05:32:57.510638 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:57.510534    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02cffa-7875b685ff-pchf5"
Oct 30 05:32:57.597702 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:57.597640    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02cffa-7875b685ff-pchf5" podStartSLOduration=12.278397669 podStartE2EDuration="13.597626078s" totalImagesPullingTime="1.319228409s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:32:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:32:57.5970199 +0000 UTC m=+2906.820839802" watchObservedRunningTime="2026-10-30 05:32:57.597626078 +0000 UTC m=+2906.821445978"
Oct 30 05:32:57.720856 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:57.720784    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:57.854929 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:57.852597    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:58.153467 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:58.153417    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:59.419321 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:59.417892    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-45607d-669bbf79bd-n7c4k"]
Oct 30 05:32:59.419321 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:59.418153    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45607d-669bbf79bd-n7c4k" podUID="dcd3bd3a-e827-4762-97b8-5d1295c48811" containerName="maas-api" containerID="cri-o://3ea304aacde8642a948c17795f82e7bb5bd90e4355036ddbc5be0c74bb8adbfd" gracePeriod=30
Oct 30 05:32:59.471972 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:59.471937    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:59.557407 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:59.557376    5898 generic.go:356] "Generic (PLEG): container finished" podID="dcd3bd3a-e827-4762-97b8-5d1295c48811" containerID="3ea304aacde8642a948c17795f82e7bb5bd90e4355036ddbc5be0c74bb8adbfd" exitCode=0
Oct 30 05:32:59.557567 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:59.557461    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45607d-669bbf79bd-n7c4k" event={"ID":"dcd3bd3a-e827-4762-97b8-5d1295c48811","Type":"ContainerDied","Data":"3ea304aacde8642a948c17795f82e7bb5bd90e4355036ddbc5be0c74bb8adbfd"}
Oct 30 05:32:59.727414 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:59.727386    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:59.744940 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:59.744914    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45607d-669bbf79bd-n7c4k"
Oct 30 05:32:59.830788 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:59.830754    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw"]
Oct 30 05:32:59.832474 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:59.832448    5898 cpu_manager.go:414] "containerMap: removing container" podUID="dcd3bd3a-e827-4762-97b8-5d1295c48811" containerName="maas-api"
Oct 30 05:32:59.832474 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:59.832475    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcd3bd3a-e827-4762-97b8-5d1295c48811" containerName="maas-api"
Oct 30 05:32:59.832730 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:59.832713    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcd3bd3a-e827-4762-97b8-5d1295c48811" containerName="maas-api"
Oct 30 05:32:59.838974 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:59.838857    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw"
Oct 30 05:32:59.844641 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:59.844251    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcd3bd3a-e827-4762-97b8-5d1295c48811-kube-api-access-52ds5\" (UniqueName: \"kubernetes.io/projected/dcd3bd3a-e827-4762-97b8-5d1295c48811-kube-api-access-52ds5\") pod \"dcd3bd3a-e827-4762-97b8-5d1295c48811\" (UID: \"dcd3bd3a-e827-4762-97b8-5d1295c48811\") "
Oct 30 05:32:59.844641 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:59.844380    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcd3bd3a-e827-4762-97b8-5d1295c48811-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcd3bd3a-e827-4762-97b8-5d1295c48811-maas-api-tls\") pod \"dcd3bd3a-e827-4762-97b8-5d1295c48811\" (UID: \"dcd3bd3a-e827-4762-97b8-5d1295c48811\") "
Oct 30 05:32:59.844641 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:59.844434    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcd3bd3a-e827-4762-97b8-5d1295c48811-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcd3bd3a-e827-4762-97b8-5d1295c48811-metrics-certs\") pod \"dcd3bd3a-e827-4762-97b8-5d1295c48811\" (UID: \"dcd3bd3a-e827-4762-97b8-5d1295c48811\") "
Oct 30 05:32:59.844641 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:59.844625    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcd3bd3a-e827-4762-97b8-5d1295c48811-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcd3bd3a-e827-4762-97b8-5d1295c48811-openshift-service-ca\") pod \"dcd3bd3a-e827-4762-97b8-5d1295c48811\" (UID: \"dcd3bd3a-e827-4762-97b8-5d1295c48811\") "
Oct 30 05:32:59.845026 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:59.845003    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/268c3238-4e5e-4907-b5c0-cf47164701ea-istio-envoy\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw\" (UID: \"268c3238-4e5e-4907-b5c0-cf47164701ea\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw"
Oct 30 05:32:59.845108 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:59.845091    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/268c3238-4e5e-4907-b5c0-cf47164701ea-istio-token\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw\" (UID: \"268c3238-4e5e-4907-b5c0-cf47164701ea\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw"
Oct 30 05:32:59.845153 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:59.845126    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/268c3238-4e5e-4907-b5c0-cf47164701ea-istio-data\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw\" (UID: \"268c3238-4e5e-4907-b5c0-cf47164701ea\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw"
Oct 30 05:32:59.845198 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:59.845177    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvhzz\" (UniqueName: \"kubernetes.io/projected/268c3238-4e5e-4907-b5c0-cf47164701ea-kube-api-access-pvhzz\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw\" (UID: \"268c3238-4e5e-4907-b5c0-cf47164701ea\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw"
Oct 30 05:32:59.845282 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:59.845264    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/268c3238-4e5e-4907-b5c0-cf47164701ea-workload-socket\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw\" (UID: \"268c3238-4e5e-4907-b5c0-cf47164701ea\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw"
Oct 30 05:32:59.846069 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:59.846043    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcd3bd3a-e827-4762-97b8-5d1295c48811-openshift-service-ca" pod "dcd3bd3a-e827-4762-97b8-5d1295c48811" (UID: "dcd3bd3a-e827-4762-97b8-5d1295c48811"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:32:59.846179 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:59.846119    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/268c3238-4e5e-4907-b5c0-cf47164701ea-istio-podinfo\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw\" (UID: \"268c3238-4e5e-4907-b5c0-cf47164701ea\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw"
Oct 30 05:32:59.846245 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:59.846202    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/268c3238-4e5e-4907-b5c0-cf47164701ea-istiod-ca-cert\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw\" (UID: \"268c3238-4e5e-4907-b5c0-cf47164701ea\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw"
Oct 30 05:32:59.846323 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:59.846295    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/268c3238-4e5e-4907-b5c0-cf47164701ea-credential-socket\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw\" (UID: \"268c3238-4e5e-4907-b5c0-cf47164701ea\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw"
Oct 30 05:32:59.846401 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:59.846385    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/268c3238-4e5e-4907-b5c0-cf47164701ea-workload-certs\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw\" (UID: \"268c3238-4e5e-4907-b5c0-cf47164701ea\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw"
Oct 30 05:32:59.846730 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:59.846707    5898 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcd3bd3a-e827-4762-97b8-5d1295c48811-openshift-service-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:32:59.849347 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:59.849228    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcd3bd3a-e827-4762-97b8-5d1295c48811-maas-api-tls" pod "dcd3bd3a-e827-4762-97b8-5d1295c48811" (UID: "dcd3bd3a-e827-4762-97b8-5d1295c48811"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:32:59.849576 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:59.849524    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcd3bd3a-e827-4762-97b8-5d1295c48811-kube-api-access-52ds5" pod "dcd3bd3a-e827-4762-97b8-5d1295c48811" (UID: "dcd3bd3a-e827-4762-97b8-5d1295c48811"). InnerVolumeSpecName "kube-api-access-52ds5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:32:59.850161 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:59.850136    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcd3bd3a-e827-4762-97b8-5d1295c48811-metrics-certs" pod "dcd3bd3a-e827-4762-97b8-5d1295c48811" (UID: "dcd3bd3a-e827-4762-97b8-5d1295c48811"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:32:59.860776 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:59.860730    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-02cffa-openshift-default-dockercfg-wdscb\""
Oct 30 05:32:59.867947 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:59.865832    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw"]
Oct 30 05:32:59.868299 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:59.868274    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:32:59.960987 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:59.960833    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/268c3238-4e5e-4907-b5c0-cf47164701ea-credential-socket\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw\" (UID: \"268c3238-4e5e-4907-b5c0-cf47164701ea\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw"
Oct 30 05:32:59.960987 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:59.960929    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/268c3238-4e5e-4907-b5c0-cf47164701ea-workload-certs\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw\" (UID: \"268c3238-4e5e-4907-b5c0-cf47164701ea\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw"
Oct 30 05:32:59.961235 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:59.961142    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/268c3238-4e5e-4907-b5c0-cf47164701ea-istio-envoy\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw\" (UID: \"268c3238-4e5e-4907-b5c0-cf47164701ea\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw"
Oct 30 05:32:59.961235 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:59.961227    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/268c3238-4e5e-4907-b5c0-cf47164701ea-istio-token\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw\" (UID: \"268c3238-4e5e-4907-b5c0-cf47164701ea\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw"
Oct 30 05:32:59.961358 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:59.961257    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/268c3238-4e5e-4907-b5c0-cf47164701ea-istio-data\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw\" (UID: \"268c3238-4e5e-4907-b5c0-cf47164701ea\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw"
Oct 30 05:32:59.961414 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:59.961342    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvhzz\" (UniqueName: \"kubernetes.io/projected/268c3238-4e5e-4907-b5c0-cf47164701ea-kube-api-access-pvhzz\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw\" (UID: \"268c3238-4e5e-4907-b5c0-cf47164701ea\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw"
Oct 30 05:32:59.961481 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:59.961469    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/268c3238-4e5e-4907-b5c0-cf47164701ea-workload-socket\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw\" (UID: \"268c3238-4e5e-4907-b5c0-cf47164701ea\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw"
Oct 30 05:32:59.961540 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:59.961511    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/268c3238-4e5e-4907-b5c0-cf47164701ea-istio-podinfo\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw\" (UID: \"268c3238-4e5e-4907-b5c0-cf47164701ea\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw"
Oct 30 05:32:59.961622 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:59.961603    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/268c3238-4e5e-4907-b5c0-cf47164701ea-istiod-ca-cert\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw\" (UID: \"268c3238-4e5e-4907-b5c0-cf47164701ea\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw"
Oct 30 05:32:59.961725 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:59.961710    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcd3bd3a-e827-4762-97b8-5d1295c48811-maas-api-tls\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:32:59.961795 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:59.961739    5898 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcd3bd3a-e827-4762-97b8-5d1295c48811-metrics-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:32:59.961795 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:59.961755    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52ds5\" (UniqueName: \"kubernetes.io/projected/dcd3bd3a-e827-4762-97b8-5d1295c48811-kube-api-access-52ds5\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:32:59.962694 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:59.962665    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/268c3238-4e5e-4907-b5c0-cf47164701ea-istiod-ca-cert\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw\" (UID: \"268c3238-4e5e-4907-b5c0-cf47164701ea\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw"
Oct 30 05:32:59.962816 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:59.962701    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/268c3238-4e5e-4907-b5c0-cf47164701ea-credential-socket\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw\" (UID: \"268c3238-4e5e-4907-b5c0-cf47164701ea\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw"
Oct 30 05:32:59.963031 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:59.963008    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/268c3238-4e5e-4907-b5c0-cf47164701ea-istio-data\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw\" (UID: \"268c3238-4e5e-4907-b5c0-cf47164701ea\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw"
Oct 30 05:32:59.963203 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:59.963021    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/268c3238-4e5e-4907-b5c0-cf47164701ea-workload-certs\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw\" (UID: \"268c3238-4e5e-4907-b5c0-cf47164701ea\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw"
Oct 30 05:32:59.963592 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:59.963571    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/268c3238-4e5e-4907-b5c0-cf47164701ea-workload-socket\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw\" (UID: \"268c3238-4e5e-4907-b5c0-cf47164701ea\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw"
Oct 30 05:32:59.966263 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:59.966242    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/268c3238-4e5e-4907-b5c0-cf47164701ea-istio-envoy\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw\" (UID: \"268c3238-4e5e-4907-b5c0-cf47164701ea\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw"
Oct 30 05:32:59.992082 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:32:59.992052    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/268c3238-4e5e-4907-b5c0-cf47164701ea-istio-podinfo\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw\" (UID: \"268c3238-4e5e-4907-b5c0-cf47164701ea\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw"
Oct 30 05:33:00.012827 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:00.012792    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvhzz\" (UniqueName: \"kubernetes.io/projected/268c3238-4e5e-4907-b5c0-cf47164701ea-kube-api-access-pvhzz\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw\" (UID: \"268c3238-4e5e-4907-b5c0-cf47164701ea\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw"
Oct 30 05:33:00.015310 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:00.015253    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/268c3238-4e5e-4907-b5c0-cf47164701ea-istio-token\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw\" (UID: \"268c3238-4e5e-4907-b5c0-cf47164701ea\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw"
Oct 30 05:33:00.179418 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:00.179385    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw"
Oct 30 05:33:00.419542 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:00.419490    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:00.568916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:00.568818    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45607d-669bbf79bd-n7c4k"
Oct 30 05:33:00.568916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:00.568849    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45607d-669bbf79bd-n7c4k" event={"ID":"dcd3bd3a-e827-4762-97b8-5d1295c48811","Type":"ContainerDied","Data":"8d326ef6f772e216d89d8857c7dd401fdbbc9c4fb8f7dbe2a51b1b4d8cf59fb6"}
Oct 30 05:33:00.569140 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:00.568946    5898 scope.go:122] "RemoveContainer" containerID="3ea304aacde8642a948c17795f82e7bb5bd90e4355036ddbc5be0c74bb8adbfd"
Oct 30 05:33:00.584503 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:00.583716    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:00.650293 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:33:00.650259    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod268c3238_4e5e_4907_b5c0_cf47164701ea.slice/crio-23bbc9e7d19c021592534be6b6084770e666975402f759e97d5cf43b1b8cb7c0 WatchSource:0}: Error finding container 23bbc9e7d19c021592534be6b6084770e666975402f759e97d5cf43b1b8cb7c0: Status 404 returned error can't find the container with id 23bbc9e7d19c021592534be6b6084770e666975402f759e97d5cf43b1b8cb7c0
Oct 30 05:33:00.654640 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:00.654602    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:33:00.654732 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:00.654689    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:33:00.654805 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:00.654731    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:33:00.659821 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:00.659796    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw"]
Oct 30 05:33:00.664134 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:00.664100    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45607d-5d4xh"]
Oct 30 05:33:00.667662 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:00.667637    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-45607d-5d4xh"]
Oct 30 05:33:00.674047 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:00.674025    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-45607d-669bbf79bd-n7c4k"]
Oct 30 05:33:00.677890 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:00.677739    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-45607d-669bbf79bd-n7c4k"]
Oct 30 05:33:00.704961 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:00.704935    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:00.896924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:00.894971    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcd3bd3a-e827-4762-97b8-5d1295c48811" path="/var/lib/kubelet/pods/dcd3bd3a-e827-4762-97b8-5d1295c48811/volumes"
Oct 30 05:33:00.914082 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:00.914049    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e34f763e-383c-4de4-921b-416c969ec6f0" path="/var/lib/kubelet/pods/e34f763e-383c-4de4-921b-416c969ec6f0/volumes"
Oct 30 05:33:01.129700 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:01.129665    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:01.579879 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:01.579836    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw" event={"ID":"268c3238-4e5e-4907-b5c0-cf47164701ea","Type":"ContainerStarted","Data":"18fece5dd1ef979c4b12e33f0b095af6f204fe0d53079458da9f7ca662fb76f4"}
Oct 30 05:33:01.580388 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:01.579955    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw" event={"ID":"268c3238-4e5e-4907-b5c0-cf47164701ea","Type":"ContainerStarted","Data":"23bbc9e7d19c021592534be6b6084770e666975402f759e97d5cf43b1b8cb7c0"}
Oct 30 05:33:01.652458 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:01.652405    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw" podStartSLOduration=6.652392536 podStartE2EDuration="6.652392536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:32:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:33:01.639434766 +0000 UTC m=+2910.863254667" watchObservedRunningTime="2026-10-30 05:33:01.652392536 +0000 UTC m=+2910.876212437"
Oct 30 05:33:02.198928 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:02.198283    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw"
Oct 30 05:33:02.200608 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:02.200573    5898 patch_prober.go:29] interesting pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 05:33:02.200760 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:02.200636    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw" podUID="268c3238-4e5e-4907-b5c0-cf47164701ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 05:33:02.745651 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:02.745621    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:03.196925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:03.195317    5898 patch_prober.go:29] interesting pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 05:33:03.196925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:03.195383    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw" podUID="268c3238-4e5e-4907-b5c0-cf47164701ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 05:33:04.192939 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:04.188256    5898 patch_prober.go:29] interesting pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 05:33:04.192939 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:04.188316    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw" podUID="268c3238-4e5e-4907-b5c0-cf47164701ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 05:33:05.246924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:05.224229    5898 patch_prober.go:29] interesting pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 05:33:05.246924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:05.224655    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw" podUID="268c3238-4e5e-4907-b5c0-cf47164701ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 05:33:05.371925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:05.371252    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6" podUID="47ff272a-ac5c-4f59-bf7b-5c0692d09386" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Oct 30 05:33:05.465929 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:05.463835    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6" podUID="b09804e3-8135-4f6d-ae34-c45d6da9728e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:33:05.539978 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:05.536958    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh" podUID="38428a53-04c7-400a-9c78-52407e982478" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:33:05.548928 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:05.542947    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d"]
Oct 30 05:33:05.548928 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:05.543305    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d" podUID="572f0bd4-2d12-4bcb-adce-edf0c3926179" containerName="istio-proxy" containerID="cri-o://ab62b30ea1c501479ed1b4c44fbd0a12657a9d555f28f355fb57561da7b0ac77" gracePeriod=30
Oct 30 05:33:05.740682 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:05.642415    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx" podUID="d523008f-c740-4a41-89cb-e82e1ffa0094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 05:33:05.759363 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:05.759315    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5" podUID="ebf8f4a2-8ac6-48ec-bdba-3f6a99653264" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 05:33:05.820469 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:05.819974    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt"]
Oct 30 05:33:05.820469 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:05.820311    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt" podUID="b649a15a-bda6-4ad5-852d-8c4253f5900f" containerName="istio-proxy" containerID="cri-o://4b53150c7c4665de19177d1eeb2da6a31b842ced080e90e96f8b81776a5eb0c8" gracePeriod=30
Oct 30 05:33:05.820862 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:05.820826    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g" podUID="70294a85-1050-4632-b3bf-c3e44a4c0b41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:33:05.898579 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:05.874135    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247" podUID="1d865d3e-a6e9-407e-844e-cc75fc7f1354" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:33:05.992719 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:05.992676    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9" podUID="1647f909-5580-4219-a2e1-30b4cd994379" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:33:06.037025 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:06.036961    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:06.052175 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:06.046480    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849" podUID="a9707f50-d5e1-409f-b8f3-c58629677aaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 05:33:06.070112 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:06.070066    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49" podUID="05eef4eb-f379-4dc7-a947-123bd85a4968" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 05:33:06.125058 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:06.125010    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6" podUID="d30a7a86-f1da-451c-81dd-ebdbaeb4f8af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:33:06.225959 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:06.214765    5898 patch_prober.go:29] interesting pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 05:33:06.225959 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:06.215073    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw" podUID="268c3238-4e5e-4907-b5c0-cf47164701ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 05:33:06.225959 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:06.219730    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/distinct-w5-0b21ba-kserve-7674c59666-m2sbd" podUID="24c6b2d4-3377-4079-8273-3f48064c3a98" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:33:06.229922 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:06.229537    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:06.264074 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:06.264031    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58" podUID="c27f3b21-1447-4af9-8d99-c04fa9b23cda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:33:06.288935 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:06.281472    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb" podUID="6f16bd6a-a7f0-49bb-844f-57b777bde2ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:33:06.378186 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:06.378061    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6" podUID="47ff272a-ac5c-4f59-bf7b-5c0692d09386" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:33:06.379346 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:06.379327    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6"
Oct 30 05:33:06.510925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:06.496065    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6" podUID="b09804e3-8135-4f6d-ae34-c45d6da9728e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:33:06.510925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:06.500131    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:06.510925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:06.505405    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6"
Oct 30 05:33:06.552015 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:06.551973    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh" podUID="38428a53-04c7-400a-9c78-52407e982478" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:33:06.567931 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:06.560007    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh"
Oct 30 05:33:06.654340 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:06.654236    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx" podUID="d523008f-c740-4a41-89cb-e82e1ffa0094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:33:06.677460 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:06.677431    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx"
Oct 30 05:33:06.825716 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:06.825668    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g" podUID="70294a85-1050-4632-b3bf-c3e44a4c0b41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:33:06.837708 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:06.837681    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g"
Oct 30 05:33:06.862296 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:06.862086    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5" podUID="ebf8f4a2-8ac6-48ec-bdba-3f6a99653264" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:33:06.862848 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:06.862818    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5"
Oct 30 05:33:06.890322 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:06.890287    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247" podUID="1d865d3e-a6e9-407e-844e-cc75fc7f1354" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:33:06.891916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:06.891882    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247"
Oct 30 05:33:07.006777 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:07.002167    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9" podUID="1647f909-5580-4219-a2e1-30b4cd994379" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:33:07.071923 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:07.070081    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849" podUID="a9707f50-d5e1-409f-b8f3-c58629677aaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:33:07.108866 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:07.072369    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49" podUID="05eef4eb-f379-4dc7-a947-123bd85a4968" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:33:07.151308 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:07.151263    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6" podUID="d30a7a86-f1da-451c-81dd-ebdbaeb4f8af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:33:07.368813 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:07.230030    5898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58" podUID="c27f3b21-1447-4af9-8d99-c04fa9b23cda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": context deadline exceeded"
Oct 30 05:33:07.369404 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:07.250303    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/distinct-w5-0b21ba-kserve-7674c59666-m2sbd" podUID="24c6b2d4-3377-4079-8273-3f48064c3a98" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": context deadline exceeded"
Oct 30 05:33:07.371974 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:07.259052    5898 patch_prober.go:29] interesting pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 05:33:07.371974 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:07.370810    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw" podUID="268c3238-4e5e-4907-b5c0-cf47164701ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 05:33:07.371974 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:07.305592    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb" podUID="6f16bd6a-a7f0-49bb-844f-57b777bde2ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 05:33:07.373010 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:07.305524    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:07.373225 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:07.346219    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h"]
Oct 30 05:33:07.397936 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:07.373413    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9"
Oct 30 05:33:07.449785 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:07.439345    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849"
Oct 30 05:33:07.449785 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:07.439394    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49"
Oct 30 05:33:07.449785 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:07.439424    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9"
Oct 30 05:33:07.449785 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:07.439446    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx"
Oct 30 05:33:07.449785 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:07.439464    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849"
Oct 30 05:33:07.449785 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:07.439492    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247"
Oct 30 05:33:07.449785 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:07.439517    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0b21ba/distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6"
Oct 30 05:33:07.449785 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:07.439539    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0b21ba/distinct-w5-0b21ba-kserve-7674c59666-m2sbd"
Oct 30 05:33:07.449785 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:07.439575    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb"
Oct 30 05:33:07.452438 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:07.306340    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58" podUID="c27f3b21-1447-4af9-8d99-c04fa9b23cda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:33:07.456132 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:07.382231    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h" podUID="6c2a73f8-80fb-46e4-ac53-50ca8ed027d9" containerName="istio-proxy" containerID="cri-o://80ffd4c0bfdb6404bbba58983a4fe831f9ca4e0c7a4e4880c200434081160f34" gracePeriod=30
Oct 30 05:33:07.456132 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:07.382322    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6" podUID="47ff272a-ac5c-4f59-bf7b-5c0692d09386" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:33:07.458225 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:07.458203    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58"
Oct 30 05:33:07.480935 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:07.477589    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh"
Oct 30 05:33:07.528480 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:07.526063    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5"
Oct 30 05:33:07.528480 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:07.527012    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6" podUID="b09804e3-8135-4f6d-ae34-c45d6da9728e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:33:07.543891 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:07.543861    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g"
Oct 30 05:33:07.573549 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:07.573519    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0b21ba/distinct-w5-0b21ba-kserve-7674c59666-m2sbd"
Oct 30 05:33:07.601987 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:07.601960    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb"
Oct 30 05:33:07.659981 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:07.651382    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49"
Oct 30 05:33:07.757433 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:07.757281    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58"
Oct 30 05:33:07.766208 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:07.766177    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6"
Oct 30 05:33:07.766377 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:07.766253    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6"
Oct 30 05:33:07.768119 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:07.768097    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0b21ba/distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6"
Oct 30 05:33:08.204122 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:08.204084    5898 patch_prober.go:29] interesting pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 05:33:08.204310 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:08.204163    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw" podUID="268c3238-4e5e-4907-b5c0-cf47164701ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 05:33:08.558809 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:08.558728    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02cffa-7875b685ff-pchf5"
Oct 30 05:33:08.966074 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:08.965615    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:09.189511 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:09.189477    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw"
Oct 30 05:33:09.189959 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:09.189938    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw"
Oct 30 05:33:09.191257 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:09.191236    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw"
Oct 30 05:33:10.372203 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:10.372173    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:10.487617 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:10.487560    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:10.840978 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:10.840932    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2"]
Oct 30 05:33:10.841243 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:10.841221    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2" podUID="cb8ee7a6-2a74-411a-8985-c2422f295292" containerName="istio-proxy" containerID="cri-o://dab4512284b5d8df32e9e348f6d2614ee855a106c6a3ef4c763b396cc3c1a072" gracePeriod=30
Oct 30 05:33:10.860597 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:10.860571    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:11.036350 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.036272    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:11.493896 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.493867    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d"
Oct 30 05:33:11.526312 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.526285    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/572f0bd4-2d12-4bcb-adce-edf0c3926179-istio-token\" (UniqueName: \"kubernetes.io/projected/572f0bd4-2d12-4bcb-adce-edf0c3926179-istio-token\") pod \"572f0bd4-2d12-4bcb-adce-edf0c3926179\" (UID: \"572f0bd4-2d12-4bcb-adce-edf0c3926179\") "
Oct 30 05:33:11.526442 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.526356    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/572f0bd4-2d12-4bcb-adce-edf0c3926179-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/572f0bd4-2d12-4bcb-adce-edf0c3926179-credential-socket\") pod \"572f0bd4-2d12-4bcb-adce-edf0c3926179\" (UID: \"572f0bd4-2d12-4bcb-adce-edf0c3926179\") "
Oct 30 05:33:11.526442 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.526404    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/572f0bd4-2d12-4bcb-adce-edf0c3926179-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/572f0bd4-2d12-4bcb-adce-edf0c3926179-istiod-ca-cert\") pod \"572f0bd4-2d12-4bcb-adce-edf0c3926179\" (UID: \"572f0bd4-2d12-4bcb-adce-edf0c3926179\") "
Oct 30 05:33:11.526532 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.526490    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/572f0bd4-2d12-4bcb-adce-edf0c3926179-kube-api-access-jxhn6\" (UniqueName: \"kubernetes.io/projected/572f0bd4-2d12-4bcb-adce-edf0c3926179-kube-api-access-jxhn6\") pod \"572f0bd4-2d12-4bcb-adce-edf0c3926179\" (UID: \"572f0bd4-2d12-4bcb-adce-edf0c3926179\") "
Oct 30 05:33:11.526570 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.526533    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/572f0bd4-2d12-4bcb-adce-edf0c3926179-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/572f0bd4-2d12-4bcb-adce-edf0c3926179-istio-envoy\") pod \"572f0bd4-2d12-4bcb-adce-edf0c3926179\" (UID: \"572f0bd4-2d12-4bcb-adce-edf0c3926179\") "
Oct 30 05:33:11.526570 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.526561    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/572f0bd4-2d12-4bcb-adce-edf0c3926179-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/572f0bd4-2d12-4bcb-adce-edf0c3926179-workload-socket\") pod \"572f0bd4-2d12-4bcb-adce-edf0c3926179\" (UID: \"572f0bd4-2d12-4bcb-adce-edf0c3926179\") "
Oct 30 05:33:11.526633 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.526604    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/572f0bd4-2d12-4bcb-adce-edf0c3926179-credential-socket" pod "572f0bd4-2d12-4bcb-adce-edf0c3926179" (UID: "572f0bd4-2d12-4bcb-adce-edf0c3926179"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:33:11.526669 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.526654    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/572f0bd4-2d12-4bcb-adce-edf0c3926179-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/572f0bd4-2d12-4bcb-adce-edf0c3926179-workload-certs\") pod \"572f0bd4-2d12-4bcb-adce-edf0c3926179\" (UID: \"572f0bd4-2d12-4bcb-adce-edf0c3926179\") "
Oct 30 05:33:11.526704 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.526686    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/572f0bd4-2d12-4bcb-adce-edf0c3926179-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/572f0bd4-2d12-4bcb-adce-edf0c3926179-istio-data\") pod \"572f0bd4-2d12-4bcb-adce-edf0c3926179\" (UID: \"572f0bd4-2d12-4bcb-adce-edf0c3926179\") "
Oct 30 05:33:11.526736 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.526715    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/572f0bd4-2d12-4bcb-adce-edf0c3926179-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/572f0bd4-2d12-4bcb-adce-edf0c3926179-istio-podinfo\") pod \"572f0bd4-2d12-4bcb-adce-edf0c3926179\" (UID: \"572f0bd4-2d12-4bcb-adce-edf0c3926179\") "
Oct 30 05:33:11.527114 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.527089    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/572f0bd4-2d12-4bcb-adce-edf0c3926179-workload-socket" pod "572f0bd4-2d12-4bcb-adce-edf0c3926179" (UID: "572f0bd4-2d12-4bcb-adce-edf0c3926179"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:33:11.527218 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.527133    5898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/572f0bd4-2d12-4bcb-adce-edf0c3926179-credential-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:33:11.527355 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.527331    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/572f0bd4-2d12-4bcb-adce-edf0c3926179-workload-certs" pod "572f0bd4-2d12-4bcb-adce-edf0c3926179" (UID: "572f0bd4-2d12-4bcb-adce-edf0c3926179"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:33:11.527493 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.527467    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/572f0bd4-2d12-4bcb-adce-edf0c3926179-istio-data" pod "572f0bd4-2d12-4bcb-adce-edf0c3926179" (UID: "572f0bd4-2d12-4bcb-adce-edf0c3926179"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:33:11.527571 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.527474    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/572f0bd4-2d12-4bcb-adce-edf0c3926179-istiod-ca-cert" pod "572f0bd4-2d12-4bcb-adce-edf0c3926179" (UID: "572f0bd4-2d12-4bcb-adce-edf0c3926179"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:33:11.528551 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.528520    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/572f0bd4-2d12-4bcb-adce-edf0c3926179-istio-token" pod "572f0bd4-2d12-4bcb-adce-edf0c3926179" (UID: "572f0bd4-2d12-4bcb-adce-edf0c3926179"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:33:11.528701 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.528676    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/572f0bd4-2d12-4bcb-adce-edf0c3926179-kube-api-access-jxhn6" pod "572f0bd4-2d12-4bcb-adce-edf0c3926179" (UID: "572f0bd4-2d12-4bcb-adce-edf0c3926179"). InnerVolumeSpecName "kube-api-access-jxhn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:33:11.529141 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.529119    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/572f0bd4-2d12-4bcb-adce-edf0c3926179-istio-envoy" pod "572f0bd4-2d12-4bcb-adce-edf0c3926179" (UID: "572f0bd4-2d12-4bcb-adce-edf0c3926179"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:33:11.529230 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.529213    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/572f0bd4-2d12-4bcb-adce-edf0c3926179-istio-podinfo" pod "572f0bd4-2d12-4bcb-adce-edf0c3926179" (UID: "572f0bd4-2d12-4bcb-adce-edf0c3926179"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:33:11.529475 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.529452    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt"
Oct 30 05:33:11.555421 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.555384    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k"]
Oct 30 05:33:11.556553 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.556524    5898 cpu_manager.go:414] "containerMap: removing container" podUID="b649a15a-bda6-4ad5-852d-8c4253f5900f" containerName="istio-proxy"
Oct 30 05:33:11.556553 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.556551    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b649a15a-bda6-4ad5-852d-8c4253f5900f" containerName="istio-proxy"
Oct 30 05:33:11.556748 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.556598    5898 cpu_manager.go:414] "containerMap: removing container" podUID="572f0bd4-2d12-4bcb-adce-edf0c3926179" containerName="istio-proxy"
Oct 30 05:33:11.556748 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.556606    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="572f0bd4-2d12-4bcb-adce-edf0c3926179" containerName="istio-proxy"
Oct 30 05:33:11.556855 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.556782    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="572f0bd4-2d12-4bcb-adce-edf0c3926179" containerName="istio-proxy"
Oct 30 05:33:11.556855 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.556813    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="b649a15a-bda6-4ad5-852d-8c4253f5900f" containerName="istio-proxy"
Oct 30 05:33:11.561362 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.561341    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k"
Oct 30 05:33:11.589863 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.589833    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k"]
Oct 30 05:33:11.603065 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.603034    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-02cffa-858d55bbb-hxc8w"]
Oct 30 05:33:11.609592 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.609572    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-02cffa-858d55bbb-hxc8w"
Oct 30 05:33:11.623929 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.619506    5898 generic.go:356] "Generic (PLEG): container finished" podID="572f0bd4-2d12-4bcb-adce-edf0c3926179" containerID="ab62b30ea1c501479ed1b4c44fbd0a12657a9d555f28f355fb57561da7b0ac77" exitCode=0
Oct 30 05:33:11.623929 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.619796    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d"
Oct 30 05:33:11.623929 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.620552    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d" event={"ID":"572f0bd4-2d12-4bcb-adce-edf0c3926179","Type":"ContainerDied","Data":"ab62b30ea1c501479ed1b4c44fbd0a12657a9d555f28f355fb57561da7b0ac77"}
Oct 30 05:33:11.623929 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.620603    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d" event={"ID":"572f0bd4-2d12-4bcb-adce-edf0c3926179","Type":"ContainerDied","Data":"aaf7bd51bedb39f040a68b04140f9fba53a3a4bb47ec603ba4f4eae36b205e73"}
Oct 30 05:33:11.623929 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.620631    5898 scope.go:122] "RemoveContainer" containerID="ab62b30ea1c501479ed1b4c44fbd0a12657a9d555f28f355fb57561da7b0ac77"
Oct 30 05:33:11.623929 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.621591    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-02cffa-858d55bbb-hxc8w"]
Oct 30 05:33:11.627846 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.627821    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b649a15a-bda6-4ad5-852d-8c4253f5900f-istio-token\" (UniqueName: \"kubernetes.io/projected/b649a15a-bda6-4ad5-852d-8c4253f5900f-istio-token\") pod \"b649a15a-bda6-4ad5-852d-8c4253f5900f\" (UID: \"b649a15a-bda6-4ad5-852d-8c4253f5900f\") "
Oct 30 05:33:11.627968 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.627886    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b649a15a-bda6-4ad5-852d-8c4253f5900f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b649a15a-bda6-4ad5-852d-8c4253f5900f-istio-podinfo\") pod \"b649a15a-bda6-4ad5-852d-8c4253f5900f\" (UID: \"b649a15a-bda6-4ad5-852d-8c4253f5900f\") "
Oct 30 05:33:11.628051 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.627965    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b649a15a-bda6-4ad5-852d-8c4253f5900f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b649a15a-bda6-4ad5-852d-8c4253f5900f-workload-certs\") pod \"b649a15a-bda6-4ad5-852d-8c4253f5900f\" (UID: \"b649a15a-bda6-4ad5-852d-8c4253f5900f\") "
Oct 30 05:33:11.628051 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.628017    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b649a15a-bda6-4ad5-852d-8c4253f5900f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b649a15a-bda6-4ad5-852d-8c4253f5900f-istio-data\") pod \"b649a15a-bda6-4ad5-852d-8c4253f5900f\" (UID: \"b649a15a-bda6-4ad5-852d-8c4253f5900f\") "
Oct 30 05:33:11.628163 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.628082    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b649a15a-bda6-4ad5-852d-8c4253f5900f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b649a15a-bda6-4ad5-852d-8c4253f5900f-workload-socket\") pod \"b649a15a-bda6-4ad5-852d-8c4253f5900f\" (UID: \"b649a15a-bda6-4ad5-852d-8c4253f5900f\") "
Oct 30 05:33:11.628163 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.628127    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b649a15a-bda6-4ad5-852d-8c4253f5900f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b649a15a-bda6-4ad5-852d-8c4253f5900f-istiod-ca-cert\") pod \"b649a15a-bda6-4ad5-852d-8c4253f5900f\" (UID: \"b649a15a-bda6-4ad5-852d-8c4253f5900f\") "
Oct 30 05:33:11.628665 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.628597    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b649a15a-bda6-4ad5-852d-8c4253f5900f-istio-data" pod "b649a15a-bda6-4ad5-852d-8c4253f5900f" (UID: "b649a15a-bda6-4ad5-852d-8c4253f5900f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:33:11.629016 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.628948    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b649a15a-bda6-4ad5-852d-8c4253f5900f-workload-socket" pod "b649a15a-bda6-4ad5-852d-8c4253f5900f" (UID: "b649a15a-bda6-4ad5-852d-8c4253f5900f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:33:11.629112 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.629041    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b649a15a-bda6-4ad5-852d-8c4253f5900f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b649a15a-bda6-4ad5-852d-8c4253f5900f-istio-envoy\") pod \"b649a15a-bda6-4ad5-852d-8c4253f5900f\" (UID: \"b649a15a-bda6-4ad5-852d-8c4253f5900f\") "
Oct 30 05:33:11.629112 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.629084    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b649a15a-bda6-4ad5-852d-8c4253f5900f-kube-api-access-sfwng\" (UniqueName: \"kubernetes.io/projected/b649a15a-bda6-4ad5-852d-8c4253f5900f-kube-api-access-sfwng\") pod \"b649a15a-bda6-4ad5-852d-8c4253f5900f\" (UID: \"b649a15a-bda6-4ad5-852d-8c4253f5900f\") "
Oct 30 05:33:11.629393 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.629372    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b649a15a-bda6-4ad5-852d-8c4253f5900f-istiod-ca-cert" pod "b649a15a-bda6-4ad5-852d-8c4253f5900f" (UID: "b649a15a-bda6-4ad5-852d-8c4253f5900f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:33:11.629482 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.629464    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b649a15a-bda6-4ad5-852d-8c4253f5900f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b649a15a-bda6-4ad5-852d-8c4253f5900f-credential-socket\") pod \"b649a15a-bda6-4ad5-852d-8c4253f5900f\" (UID: \"b649a15a-bda6-4ad5-852d-8c4253f5900f\") "
Oct 30 05:33:11.629807 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.629787    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c2b56cf-6fbe-4787-86db-1692ecbb41e0-credential-socket\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k\" (UID: \"4c2b56cf-6fbe-4787-86db-1692ecbb41e0\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k"
Oct 30 05:33:11.629888 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.629827    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c2b56cf-6fbe-4787-86db-1692ecbb41e0-workload-certs\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k\" (UID: \"4c2b56cf-6fbe-4787-86db-1692ecbb41e0\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k"
Oct 30 05:33:11.630009 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.629965    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c2b56cf-6fbe-4787-86db-1692ecbb41e0-istio-data\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k\" (UID: \"4c2b56cf-6fbe-4787-86db-1692ecbb41e0\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k"
Oct 30 05:33:11.630009 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.629996    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c2b56cf-6fbe-4787-86db-1692ecbb41e0-istio-token\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k\" (UID: \"4c2b56cf-6fbe-4787-86db-1692ecbb41e0\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k"
Oct 30 05:33:11.630164 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.630134    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b649a15a-bda6-4ad5-852d-8c4253f5900f-credential-socket" pod "b649a15a-bda6-4ad5-852d-8c4253f5900f" (UID: "b649a15a-bda6-4ad5-852d-8c4253f5900f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:33:11.630276 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.630259    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c2b56cf-6fbe-4787-86db-1692ecbb41e0-istio-envoy\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k\" (UID: \"4c2b56cf-6fbe-4787-86db-1692ecbb41e0\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k"
Oct 30 05:33:11.630337 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.630304    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c2b56cf-6fbe-4787-86db-1692ecbb41e0-istio-podinfo\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k\" (UID: \"4c2b56cf-6fbe-4787-86db-1692ecbb41e0\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k"
Oct 30 05:33:11.630411 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.630396    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm84s\" (UniqueName: \"kubernetes.io/projected/dc5b5678-4d59-41b9-857b-8c502ec7eb28-kube-api-access-nm84s\") pod \"payload-processing-e2e-ipp-a-02cffa-858d55bbb-hxc8w\" (UID: \"dc5b5678-4d59-41b9-857b-8c502ec7eb28\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-02cffa-858d55bbb-hxc8w"
Oct 30 05:33:11.630467 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.630449    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwjlb\" (UniqueName: \"kubernetes.io/projected/4c2b56cf-6fbe-4787-86db-1692ecbb41e0-kube-api-access-pwjlb\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k\" (UID: \"4c2b56cf-6fbe-4787-86db-1692ecbb41e0\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k"
Oct 30 05:33:11.630520 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.630478    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c2b56cf-6fbe-4787-86db-1692ecbb41e0-workload-socket\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k\" (UID: \"4c2b56cf-6fbe-4787-86db-1692ecbb41e0\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k"
Oct 30 05:33:11.630809 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.630647    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c2b56cf-6fbe-4787-86db-1692ecbb41e0-istiod-ca-cert\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k\" (UID: \"4c2b56cf-6fbe-4787-86db-1692ecbb41e0\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k"
Oct 30 05:33:11.630809 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.630675    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc5b5678-4d59-41b9-857b-8c502ec7eb28-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-02cffa-858d55bbb-hxc8w\" (UID: \"dc5b5678-4d59-41b9-857b-8c502ec7eb28\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-02cffa-858d55bbb-hxc8w"
Oct 30 05:33:11.630809 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.630763    5898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b649a15a-bda6-4ad5-852d-8c4253f5900f-workload-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:33:11.630809 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.630781    5898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b649a15a-bda6-4ad5-852d-8c4253f5900f-istiod-ca-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:33:11.630809 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.630795    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxhn6\" (UniqueName: \"kubernetes.io/projected/572f0bd4-2d12-4bcb-adce-edf0c3926179-kube-api-access-jxhn6\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:33:11.630809 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.630809    5898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/572f0bd4-2d12-4bcb-adce-edf0c3926179-istio-envoy\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:33:11.631200 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.630822    5898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/572f0bd4-2d12-4bcb-adce-edf0c3926179-workload-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:33:11.631200 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.630836    5898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b649a15a-bda6-4ad5-852d-8c4253f5900f-credential-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:33:11.631200 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.630850    5898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/572f0bd4-2d12-4bcb-adce-edf0c3926179-workload-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:33:11.631200 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.630862    5898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/572f0bd4-2d12-4bcb-adce-edf0c3926179-istio-data\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:33:11.631200 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.630875    5898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/572f0bd4-2d12-4bcb-adce-edf0c3926179-istio-podinfo\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:33:11.631200 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.630887    5898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/572f0bd4-2d12-4bcb-adce-edf0c3926179-istio-token\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:33:11.631200 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.630914    5898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/572f0bd4-2d12-4bcb-adce-edf0c3926179-istiod-ca-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:33:11.631200 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.630927    5898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b649a15a-bda6-4ad5-852d-8c4253f5900f-istio-data\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:33:11.631569 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.631225    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b649a15a-bda6-4ad5-852d-8c4253f5900f-workload-certs" pod "b649a15a-bda6-4ad5-852d-8c4253f5900f" (UID: "b649a15a-bda6-4ad5-852d-8c4253f5900f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:33:11.632138 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.632092    5898 generic.go:356] "Generic (PLEG): container finished" podID="b649a15a-bda6-4ad5-852d-8c4253f5900f" containerID="4b53150c7c4665de19177d1eeb2da6a31b842ced080e90e96f8b81776a5eb0c8" exitCode=0
Oct 30 05:33:11.632258 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.632211    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt" event={"ID":"b649a15a-bda6-4ad5-852d-8c4253f5900f","Type":"ContainerDied","Data":"4b53150c7c4665de19177d1eeb2da6a31b842ced080e90e96f8b81776a5eb0c8"}
Oct 30 05:33:11.632352 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.632256    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt" event={"ID":"b649a15a-bda6-4ad5-852d-8c4253f5900f","Type":"ContainerDied","Data":"bb9d1ec260c5c4dd80e4202505fd625167b9c308b360e66d556a837bb1dd9280"}
Oct 30 05:33:11.632352 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.632335    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt"
Oct 30 05:33:11.633365 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.633325    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b649a15a-bda6-4ad5-852d-8c4253f5900f-kube-api-access-sfwng" pod "b649a15a-bda6-4ad5-852d-8c4253f5900f" (UID: "b649a15a-bda6-4ad5-852d-8c4253f5900f"). InnerVolumeSpecName "kube-api-access-sfwng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:33:11.635847 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.635780    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b649a15a-bda6-4ad5-852d-8c4253f5900f-istio-token" pod "b649a15a-bda6-4ad5-852d-8c4253f5900f" (UID: "b649a15a-bda6-4ad5-852d-8c4253f5900f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:33:11.637386 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.637191    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b649a15a-bda6-4ad5-852d-8c4253f5900f-istio-podinfo" pod "b649a15a-bda6-4ad5-852d-8c4253f5900f" (UID: "b649a15a-bda6-4ad5-852d-8c4253f5900f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:33:11.637493 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.637414    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b649a15a-bda6-4ad5-852d-8c4253f5900f-istio-envoy" pod "b649a15a-bda6-4ad5-852d-8c4253f5900f" (UID: "b649a15a-bda6-4ad5-852d-8c4253f5900f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:33:11.645718 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.645700    5898 scope.go:122] "RemoveContainer" containerID="ab62b30ea1c501479ed1b4c44fbd0a12657a9d555f28f355fb57561da7b0ac77"
Oct 30 05:33:11.646053 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:33:11.646031    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab62b30ea1c501479ed1b4c44fbd0a12657a9d555f28f355fb57561da7b0ac77\": container with ID starting with ab62b30ea1c501479ed1b4c44fbd0a12657a9d555f28f355fb57561da7b0ac77 not found: ID does not exist" containerID="ab62b30ea1c501479ed1b4c44fbd0a12657a9d555f28f355fb57561da7b0ac77"
Oct 30 05:33:11.646150 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.646059    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab62b30ea1c501479ed1b4c44fbd0a12657a9d555f28f355fb57561da7b0ac77"} err="failed to get container status \"ab62b30ea1c501479ed1b4c44fbd0a12657a9d555f28f355fb57561da7b0ac77\": rpc error: code = NotFound desc = could not find container \"ab62b30ea1c501479ed1b4c44fbd0a12657a9d555f28f355fb57561da7b0ac77\": container with ID starting with ab62b30ea1c501479ed1b4c44fbd0a12657a9d555f28f355fb57561da7b0ac77 not found: ID does not exist"
Oct 30 05:33:11.646150 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.646076    5898 scope.go:122] "RemoveContainer" containerID="4b53150c7c4665de19177d1eeb2da6a31b842ced080e90e96f8b81776a5eb0c8"
Oct 30 05:33:11.656990 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.656966    5898 scope.go:122] "RemoveContainer" containerID="4b53150c7c4665de19177d1eeb2da6a31b842ced080e90e96f8b81776a5eb0c8"
Oct 30 05:33:11.657306 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:33:11.657281    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b53150c7c4665de19177d1eeb2da6a31b842ced080e90e96f8b81776a5eb0c8\": container with ID starting with 4b53150c7c4665de19177d1eeb2da6a31b842ced080e90e96f8b81776a5eb0c8 not found: ID does not exist" containerID="4b53150c7c4665de19177d1eeb2da6a31b842ced080e90e96f8b81776a5eb0c8"
Oct 30 05:33:11.657374 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.657306    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b53150c7c4665de19177d1eeb2da6a31b842ced080e90e96f8b81776a5eb0c8"} err="failed to get container status \"4b53150c7c4665de19177d1eeb2da6a31b842ced080e90e96f8b81776a5eb0c8\": rpc error: code = NotFound desc = could not find container \"4b53150c7c4665de19177d1eeb2da6a31b842ced080e90e96f8b81776a5eb0c8\": container with ID starting with 4b53150c7c4665de19177d1eeb2da6a31b842ced080e90e96f8b81776a5eb0c8 not found: ID does not exist"
Oct 30 05:33:11.731671 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.731639    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c2b56cf-6fbe-4787-86db-1692ecbb41e0-credential-socket\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k\" (UID: \"4c2b56cf-6fbe-4787-86db-1692ecbb41e0\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k"
Oct 30 05:33:11.731671 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.731677    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c2b56cf-6fbe-4787-86db-1692ecbb41e0-workload-certs\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k\" (UID: \"4c2b56cf-6fbe-4787-86db-1692ecbb41e0\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k"
Oct 30 05:33:11.731968 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.731732    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c2b56cf-6fbe-4787-86db-1692ecbb41e0-istio-data\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k\" (UID: \"4c2b56cf-6fbe-4787-86db-1692ecbb41e0\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k"
Oct 30 05:33:11.731968 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.731750    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c2b56cf-6fbe-4787-86db-1692ecbb41e0-istio-token\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k\" (UID: \"4c2b56cf-6fbe-4787-86db-1692ecbb41e0\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k"
Oct 30 05:33:11.731968 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.731805    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c2b56cf-6fbe-4787-86db-1692ecbb41e0-istio-envoy\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k\" (UID: \"4c2b56cf-6fbe-4787-86db-1692ecbb41e0\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k"
Oct 30 05:33:11.731968 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.731836    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c2b56cf-6fbe-4787-86db-1692ecbb41e0-istio-podinfo\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k\" (UID: \"4c2b56cf-6fbe-4787-86db-1692ecbb41e0\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k"
Oct 30 05:33:11.731968 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.731929    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm84s\" (UniqueName: \"kubernetes.io/projected/dc5b5678-4d59-41b9-857b-8c502ec7eb28-kube-api-access-nm84s\") pod \"payload-processing-e2e-ipp-a-02cffa-858d55bbb-hxc8w\" (UID: \"dc5b5678-4d59-41b9-857b-8c502ec7eb28\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-02cffa-858d55bbb-hxc8w"
Oct 30 05:33:11.732273 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.732085    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwjlb\" (UniqueName: \"kubernetes.io/projected/4c2b56cf-6fbe-4787-86db-1692ecbb41e0-kube-api-access-pwjlb\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k\" (UID: \"4c2b56cf-6fbe-4787-86db-1692ecbb41e0\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k"
Oct 30 05:33:11.732273 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.732121    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c2b56cf-6fbe-4787-86db-1692ecbb41e0-workload-socket\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k\" (UID: \"4c2b56cf-6fbe-4787-86db-1692ecbb41e0\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k"
Oct 30 05:33:11.732273 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.732140    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c2b56cf-6fbe-4787-86db-1692ecbb41e0-credential-socket\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k\" (UID: \"4c2b56cf-6fbe-4787-86db-1692ecbb41e0\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k"
Oct 30 05:33:11.732273 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.732173    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c2b56cf-6fbe-4787-86db-1692ecbb41e0-istio-data\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k\" (UID: \"4c2b56cf-6fbe-4787-86db-1692ecbb41e0\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k"
Oct 30 05:33:11.732273 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.732248    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c2b56cf-6fbe-4787-86db-1692ecbb41e0-workload-certs\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k\" (UID: \"4c2b56cf-6fbe-4787-86db-1692ecbb41e0\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k"
Oct 30 05:33:11.732522 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.732356    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c2b56cf-6fbe-4787-86db-1692ecbb41e0-istiod-ca-cert\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k\" (UID: \"4c2b56cf-6fbe-4787-86db-1692ecbb41e0\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k"
Oct 30 05:33:11.732522 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.732385    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc5b5678-4d59-41b9-857b-8c502ec7eb28-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-02cffa-858d55bbb-hxc8w\" (UID: \"dc5b5678-4d59-41b9-857b-8c502ec7eb28\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-02cffa-858d55bbb-hxc8w"
Oct 30 05:33:11.732522 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.732445    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c2b56cf-6fbe-4787-86db-1692ecbb41e0-workload-socket\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k\" (UID: \"4c2b56cf-6fbe-4787-86db-1692ecbb41e0\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k"
Oct 30 05:33:11.732522 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.732477    5898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b649a15a-bda6-4ad5-852d-8c4253f5900f-workload-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:33:11.732522 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.732494    5898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b649a15a-bda6-4ad5-852d-8c4253f5900f-istio-envoy\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:33:11.732522 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.732512    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfwng\" (UniqueName: \"kubernetes.io/projected/b649a15a-bda6-4ad5-852d-8c4253f5900f-kube-api-access-sfwng\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:33:11.732811 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.732529    5898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b649a15a-bda6-4ad5-852d-8c4253f5900f-istio-token\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:33:11.732811 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.732542    5898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b649a15a-bda6-4ad5-852d-8c4253f5900f-istio-podinfo\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:33:11.733107 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.733078    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c2b56cf-6fbe-4787-86db-1692ecbb41e0-istiod-ca-cert\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k\" (UID: \"4c2b56cf-6fbe-4787-86db-1692ecbb41e0\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k"
Oct 30 05:33:11.733265 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.733247    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc5b5678-4d59-41b9-857b-8c502ec7eb28-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-02cffa-858d55bbb-hxc8w\" (UID: \"dc5b5678-4d59-41b9-857b-8c502ec7eb28\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-02cffa-858d55bbb-hxc8w"
Oct 30 05:33:11.734434 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.734411    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c2b56cf-6fbe-4787-86db-1692ecbb41e0-istio-envoy\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k\" (UID: \"4c2b56cf-6fbe-4787-86db-1692ecbb41e0\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k"
Oct 30 05:33:11.734537 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.734518    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c2b56cf-6fbe-4787-86db-1692ecbb41e0-istio-podinfo\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k\" (UID: \"4c2b56cf-6fbe-4787-86db-1692ecbb41e0\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k"
Oct 30 05:33:11.738935 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.738917    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d"]
Oct 30 05:33:11.744166 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.744142    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-jdt4d"]
Oct 30 05:33:11.752739 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.752716    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c2b56cf-6fbe-4787-86db-1692ecbb41e0-istio-token\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k\" (UID: \"4c2b56cf-6fbe-4787-86db-1692ecbb41e0\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k"
Oct 30 05:33:11.752979 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.752956    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwjlb\" (UniqueName: \"kubernetes.io/projected/4c2b56cf-6fbe-4787-86db-1692ecbb41e0-kube-api-access-pwjlb\") pod \"e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k\" (UID: \"4c2b56cf-6fbe-4787-86db-1692ecbb41e0\") " pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k"
Oct 30 05:33:11.753034 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.753021    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm84s\" (UniqueName: \"kubernetes.io/projected/dc5b5678-4d59-41b9-857b-8c502ec7eb28-kube-api-access-nm84s\") pod \"payload-processing-e2e-ipp-a-02cffa-858d55bbb-hxc8w\" (UID: \"dc5b5678-4d59-41b9-857b-8c502ec7eb28\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-02cffa-858d55bbb-hxc8w"
Oct 30 05:33:11.877511 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.877474    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k"
Oct 30 05:33:11.937220 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.937135    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-02cffa-858d55bbb-hxc8w"
Oct 30 05:33:11.984743 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:11.984709    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247"]
Oct 30 05:33:12.095252 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:12.095217    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt"]
Oct 30 05:33:12.117959 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:12.117923    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:12.123009 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:12.122978    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-r8bxt"]
Oct 30 05:33:12.214810 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:12.214783    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k"]
Oct 30 05:33:12.217087 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:33:12.217053    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c2b56cf_6fbe_4787_86db_1692ecbb41e0.slice/crio-a0648b8bff3d67b24a885fd6fd55c82be078bcd5d36094edcaa8c404ebdf2aad WatchSource:0}: Error finding container a0648b8bff3d67b24a885fd6fd55c82be078bcd5d36094edcaa8c404ebdf2aad: Status 404 returned error can't find the container with id a0648b8bff3d67b24a885fd6fd55c82be078bcd5d36094edcaa8c404ebdf2aad
Oct 30 05:33:12.220084 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:12.220054    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:33:12.220220 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:12.220129    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:33:12.220220 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:12.220158    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:33:12.293318 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:12.293295    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-02cffa-858d55bbb-hxc8w"]
Oct 30 05:33:12.297185 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:33:12.297160    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc5b5678_4d59_41b9_857b_8c502ec7eb28.slice/crio-3ee3500e6f2839b983f06bd6096212b64c8eaf4d4588d88c017568efae1d3e46 WatchSource:0}: Error finding container 3ee3500e6f2839b983f06bd6096212b64c8eaf4d4588d88c017568efae1d3e46: Status 404 returned error can't find the container with id 3ee3500e6f2839b983f06bd6096212b64c8eaf4d4588d88c017568efae1d3e46
Oct 30 05:33:12.640239 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:12.640203    5898 patch_prober.go:29] interesting pod/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 05:33:12.640611 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:12.640261    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2" podUID="cb8ee7a6-2a74-411a-8985-c2422f295292" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 05:33:12.647332 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:12.647300    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-02cffa-858d55bbb-hxc8w" event={"ID":"dc5b5678-4d59-41b9-857b-8c502ec7eb28","Type":"ContainerStarted","Data":"c7fa77c07b732e84a6f5c1b7b64a9af9e7909ef29b7e271a7035005e52969e44"}
Oct 30 05:33:12.647454 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:12.647354    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-02cffa-858d55bbb-hxc8w" event={"ID":"dc5b5678-4d59-41b9-857b-8c502ec7eb28","Type":"ContainerStarted","Data":"3ee3500e6f2839b983f06bd6096212b64c8eaf4d4588d88c017568efae1d3e46"}
Oct 30 05:33:12.648430 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:12.648411    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-02cffa-858d55bbb-hxc8w"
Oct 30 05:33:12.650698 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:12.650667    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k" event={"ID":"4c2b56cf-6fbe-4787-86db-1692ecbb41e0","Type":"ContainerStarted","Data":"6d5251f6de12748dd21f851055ebf9a520f34f33a590f51060356d0594dc8a64"}
Oct 30 05:33:12.650817 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:12.650720    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k" event={"ID":"4c2b56cf-6fbe-4787-86db-1692ecbb41e0","Type":"ContainerStarted","Data":"a0648b8bff3d67b24a885fd6fd55c82be078bcd5d36094edcaa8c404ebdf2aad"}
Oct 30 05:33:12.656790 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:12.656685    5898 generic.go:356] "Generic (PLEG): container finished" podID="6c2a73f8-80fb-46e4-ac53-50ca8ed027d9" containerID="80ffd4c0bfdb6404bbba58983a4fe831f9ca4e0c7a4e4880c200434081160f34" exitCode=0
Oct 30 05:33:12.656790 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:12.656736    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h" event={"ID":"6c2a73f8-80fb-46e4-ac53-50ca8ed027d9","Type":"ContainerDied","Data":"80ffd4c0bfdb6404bbba58983a4fe831f9ca4e0c7a4e4880c200434081160f34"}
Oct 30 05:33:12.727265 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:12.727151    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-02cffa-858d55bbb-hxc8w" podStartSLOduration=28.727130754 podStartE2EDuration="28.727130754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:32:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:33:12.702974585 +0000 UTC m=+2921.926794517" watchObservedRunningTime="2026-10-30 05:33:12.727130754 +0000 UTC m=+2921.950950656"
Oct 30 05:33:12.787830 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:12.787766    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k" podStartSLOduration=17.787747354 podStartE2EDuration="17.787747354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:32:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:33:12.780583738 +0000 UTC m=+2922.004403666" watchObservedRunningTime="2026-10-30 05:33:12.787747354 +0000 UTC m=+2922.011567257"
Oct 30 05:33:12.788446 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:12.788418    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h"
Oct 30 05:33:12.883925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:12.867621    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c2a73f8-80fb-46e4-ac53-50ca8ed027d9-kube-api-access-rph6m\" (UniqueName: \"kubernetes.io/projected/6c2a73f8-80fb-46e4-ac53-50ca8ed027d9-kube-api-access-rph6m\") pod \"6c2a73f8-80fb-46e4-ac53-50ca8ed027d9\" (UID: \"6c2a73f8-80fb-46e4-ac53-50ca8ed027d9\") "
Oct 30 05:33:12.883925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:12.867691    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c2a73f8-80fb-46e4-ac53-50ca8ed027d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c2a73f8-80fb-46e4-ac53-50ca8ed027d9-credential-socket\") pod \"6c2a73f8-80fb-46e4-ac53-50ca8ed027d9\" (UID: \"6c2a73f8-80fb-46e4-ac53-50ca8ed027d9\") "
Oct 30 05:33:12.883925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:12.867751    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c2a73f8-80fb-46e4-ac53-50ca8ed027d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c2a73f8-80fb-46e4-ac53-50ca8ed027d9-workload-certs\") pod \"6c2a73f8-80fb-46e4-ac53-50ca8ed027d9\" (UID: \"6c2a73f8-80fb-46e4-ac53-50ca8ed027d9\") "
Oct 30 05:33:12.883925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:12.867892    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c2a73f8-80fb-46e4-ac53-50ca8ed027d9-istio-token\" (UniqueName: \"kubernetes.io/projected/6c2a73f8-80fb-46e4-ac53-50ca8ed027d9-istio-token\") pod \"6c2a73f8-80fb-46e4-ac53-50ca8ed027d9\" (UID: \"6c2a73f8-80fb-46e4-ac53-50ca8ed027d9\") "
Oct 30 05:33:12.883925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:12.868005    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c2a73f8-80fb-46e4-ac53-50ca8ed027d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c2a73f8-80fb-46e4-ac53-50ca8ed027d9-workload-socket\") pod \"6c2a73f8-80fb-46e4-ac53-50ca8ed027d9\" (UID: \"6c2a73f8-80fb-46e4-ac53-50ca8ed027d9\") "
Oct 30 05:33:12.883925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:12.868103    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c2a73f8-80fb-46e4-ac53-50ca8ed027d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c2a73f8-80fb-46e4-ac53-50ca8ed027d9-istio-data\") pod \"6c2a73f8-80fb-46e4-ac53-50ca8ed027d9\" (UID: \"6c2a73f8-80fb-46e4-ac53-50ca8ed027d9\") "
Oct 30 05:33:12.883925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:12.868187    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6c2a73f8-80fb-46e4-ac53-50ca8ed027d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c2a73f8-80fb-46e4-ac53-50ca8ed027d9-istio-podinfo\") pod \"6c2a73f8-80fb-46e4-ac53-50ca8ed027d9\" (UID: \"6c2a73f8-80fb-46e4-ac53-50ca8ed027d9\") "
Oct 30 05:33:12.883925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:12.868218    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c2a73f8-80fb-46e4-ac53-50ca8ed027d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c2a73f8-80fb-46e4-ac53-50ca8ed027d9-istio-envoy\") pod \"6c2a73f8-80fb-46e4-ac53-50ca8ed027d9\" (UID: \"6c2a73f8-80fb-46e4-ac53-50ca8ed027d9\") "
Oct 30 05:33:12.883925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:12.868249    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c2a73f8-80fb-46e4-ac53-50ca8ed027d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c2a73f8-80fb-46e4-ac53-50ca8ed027d9-istiod-ca-cert\") pod \"6c2a73f8-80fb-46e4-ac53-50ca8ed027d9\" (UID: \"6c2a73f8-80fb-46e4-ac53-50ca8ed027d9\") "
Oct 30 05:33:12.883925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:12.870656    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c2a73f8-80fb-46e4-ac53-50ca8ed027d9-istiod-ca-cert" pod "6c2a73f8-80fb-46e4-ac53-50ca8ed027d9" (UID: "6c2a73f8-80fb-46e4-ac53-50ca8ed027d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:33:12.883925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:12.870978    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c2a73f8-80fb-46e4-ac53-50ca8ed027d9-workload-socket" pod "6c2a73f8-80fb-46e4-ac53-50ca8ed027d9" (UID: "6c2a73f8-80fb-46e4-ac53-50ca8ed027d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:33:12.883925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:12.871891    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c2a73f8-80fb-46e4-ac53-50ca8ed027d9-credential-socket" pod "6c2a73f8-80fb-46e4-ac53-50ca8ed027d9" (UID: "6c2a73f8-80fb-46e4-ac53-50ca8ed027d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:33:12.883925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:12.872119    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c2a73f8-80fb-46e4-ac53-50ca8ed027d9-workload-certs" pod "6c2a73f8-80fb-46e4-ac53-50ca8ed027d9" (UID: "6c2a73f8-80fb-46e4-ac53-50ca8ed027d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:33:12.883925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:12.872177    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c2a73f8-80fb-46e4-ac53-50ca8ed027d9-istio-data" pod "6c2a73f8-80fb-46e4-ac53-50ca8ed027d9" (UID: "6c2a73f8-80fb-46e4-ac53-50ca8ed027d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:33:12.883925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:12.882525    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs"]
Oct 30 05:33:12.883925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:12.883808    5898 cpu_manager.go:414] "containerMap: removing container" podUID="6c2a73f8-80fb-46e4-ac53-50ca8ed027d9" containerName="istio-proxy"
Oct 30 05:33:12.883925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:12.883830    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c2a73f8-80fb-46e4-ac53-50ca8ed027d9" containerName="istio-proxy"
Oct 30 05:33:12.885016 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:12.884075    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c2a73f8-80fb-46e4-ac53-50ca8ed027d9" containerName="istio-proxy"
Oct 30 05:33:12.936984 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:12.921623    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c2a73f8-80fb-46e4-ac53-50ca8ed027d9-istio-podinfo" pod "6c2a73f8-80fb-46e4-ac53-50ca8ed027d9" (UID: "6c2a73f8-80fb-46e4-ac53-50ca8ed027d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:33:12.977464 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:12.977308    5898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c2a73f8-80fb-46e4-ac53-50ca8ed027d9-workload-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:33:12.977464 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:12.977346    5898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c2a73f8-80fb-46e4-ac53-50ca8ed027d9-istio-data\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:33:12.977464 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:12.977361    5898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c2a73f8-80fb-46e4-ac53-50ca8ed027d9-istio-podinfo\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:33:12.977464 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:12.977375    5898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c2a73f8-80fb-46e4-ac53-50ca8ed027d9-istiod-ca-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:33:12.977464 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:12.977389    5898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c2a73f8-80fb-46e4-ac53-50ca8ed027d9-credential-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:33:12.977464 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:12.977403    5898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c2a73f8-80fb-46e4-ac53-50ca8ed027d9-workload-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:33:13.028969 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:13.027233    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c2a73f8-80fb-46e4-ac53-50ca8ed027d9-istio-envoy" pod "6c2a73f8-80fb-46e4-ac53-50ca8ed027d9" (UID: "6c2a73f8-80fb-46e4-ac53-50ca8ed027d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:33:13.028969 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:13.027369    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c2a73f8-80fb-46e4-ac53-50ca8ed027d9-kube-api-access-rph6m" pod "6c2a73f8-80fb-46e4-ac53-50ca8ed027d9" (UID: "6c2a73f8-80fb-46e4-ac53-50ca8ed027d9"). InnerVolumeSpecName "kube-api-access-rph6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:33:13.040093 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:13.037821    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c2a73f8-80fb-46e4-ac53-50ca8ed027d9-istio-token" pod "6c2a73f8-80fb-46e4-ac53-50ca8ed027d9" (UID: "6c2a73f8-80fb-46e4-ac53-50ca8ed027d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:33:13.044558 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:13.044514    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs"]
Oct 30 05:33:13.044938 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:13.044838    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs"
Oct 30 05:33:13.045567 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:13.045551    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k"
Oct 30 05:33:13.061969 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:13.061937    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="572f0bd4-2d12-4bcb-adce-edf0c3926179" path="/var/lib/kubelet/pods/572f0bd4-2d12-4bcb-adce-edf0c3926179/volumes"
Oct 30 05:33:13.063604 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:13.063581    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b649a15a-bda6-4ad5-852d-8c4253f5900f" path="/var/lib/kubelet/pods/b649a15a-bda6-4ad5-852d-8c4253f5900f/volumes"
Oct 30 05:33:13.078938 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:13.078890    5898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c2a73f8-80fb-46e4-ac53-50ca8ed027d9-istio-envoy\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:33:13.079071 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:13.078943    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rph6m\" (UniqueName: \"kubernetes.io/projected/6c2a73f8-80fb-46e4-ac53-50ca8ed027d9-kube-api-access-rph6m\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:33:13.079071 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:13.078957    5898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c2a73f8-80fb-46e4-ac53-50ca8ed027d9-istio-token\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:33:13.138167 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:13.138113    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:13.204963 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:13.204784    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa609b95-6fbc-4cb6-9153-97e49232e302-istio-token\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs\" (UID: \"aa609b95-6fbc-4cb6-9153-97e49232e302\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs"
Oct 30 05:33:13.204963 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:13.204838    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa609b95-6fbc-4cb6-9153-97e49232e302-istiod-ca-cert\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs\" (UID: \"aa609b95-6fbc-4cb6-9153-97e49232e302\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs"
Oct 30 05:33:13.204963 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:13.204865    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l6hl\" (UniqueName: \"kubernetes.io/projected/aa609b95-6fbc-4cb6-9153-97e49232e302-kube-api-access-9l6hl\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs\" (UID: \"aa609b95-6fbc-4cb6-9153-97e49232e302\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs"
Oct 30 05:33:13.205569 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:13.205527    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa609b95-6fbc-4cb6-9153-97e49232e302-istio-podinfo\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs\" (UID: \"aa609b95-6fbc-4cb6-9153-97e49232e302\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs"
Oct 30 05:33:13.205645 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:13.205625    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa609b95-6fbc-4cb6-9153-97e49232e302-workload-socket\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs\" (UID: \"aa609b95-6fbc-4cb6-9153-97e49232e302\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs"
Oct 30 05:33:13.205700 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:13.205685    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa609b95-6fbc-4cb6-9153-97e49232e302-istio-data\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs\" (UID: \"aa609b95-6fbc-4cb6-9153-97e49232e302\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs"
Oct 30 05:33:13.206304 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:13.205943    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa609b95-6fbc-4cb6-9153-97e49232e302-credential-socket\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs\" (UID: \"aa609b95-6fbc-4cb6-9153-97e49232e302\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs"
Oct 30 05:33:13.206304 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:13.206053    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa609b95-6fbc-4cb6-9153-97e49232e302-workload-certs\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs\" (UID: \"aa609b95-6fbc-4cb6-9153-97e49232e302\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs"
Oct 30 05:33:13.206304 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:13.206147    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa609b95-6fbc-4cb6-9153-97e49232e302-istio-envoy\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs\" (UID: \"aa609b95-6fbc-4cb6-9153-97e49232e302\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs"
Oct 30 05:33:13.255031 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:13.249933    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:13.317468 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:13.315978    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa609b95-6fbc-4cb6-9153-97e49232e302-istio-podinfo\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs\" (UID: \"aa609b95-6fbc-4cb6-9153-97e49232e302\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs"
Oct 30 05:33:13.317468 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:13.316067    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa609b95-6fbc-4cb6-9153-97e49232e302-workload-socket\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs\" (UID: \"aa609b95-6fbc-4cb6-9153-97e49232e302\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs"
Oct 30 05:33:13.317468 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:13.316126    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa609b95-6fbc-4cb6-9153-97e49232e302-istio-data\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs\" (UID: \"aa609b95-6fbc-4cb6-9153-97e49232e302\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs"
Oct 30 05:33:13.317468 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:13.316182    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa609b95-6fbc-4cb6-9153-97e49232e302-credential-socket\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs\" (UID: \"aa609b95-6fbc-4cb6-9153-97e49232e302\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs"
Oct 30 05:33:13.317468 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:13.316265    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa609b95-6fbc-4cb6-9153-97e49232e302-workload-certs\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs\" (UID: \"aa609b95-6fbc-4cb6-9153-97e49232e302\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs"
Oct 30 05:33:13.317468 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:13.316322    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa609b95-6fbc-4cb6-9153-97e49232e302-istio-envoy\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs\" (UID: \"aa609b95-6fbc-4cb6-9153-97e49232e302\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs"
Oct 30 05:33:13.317468 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:13.316413    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa609b95-6fbc-4cb6-9153-97e49232e302-istio-token\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs\" (UID: \"aa609b95-6fbc-4cb6-9153-97e49232e302\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs"
Oct 30 05:33:13.317468 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:13.316442    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa609b95-6fbc-4cb6-9153-97e49232e302-istiod-ca-cert\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs\" (UID: \"aa609b95-6fbc-4cb6-9153-97e49232e302\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs"
Oct 30 05:33:13.317468 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:13.316466    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l6hl\" (UniqueName: \"kubernetes.io/projected/aa609b95-6fbc-4cb6-9153-97e49232e302-kube-api-access-9l6hl\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs\" (UID: \"aa609b95-6fbc-4cb6-9153-97e49232e302\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs"
Oct 30 05:33:13.319865 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:13.318518    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa609b95-6fbc-4cb6-9153-97e49232e302-workload-certs\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs\" (UID: \"aa609b95-6fbc-4cb6-9153-97e49232e302\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs"
Oct 30 05:33:13.319865 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:13.318784    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa609b95-6fbc-4cb6-9153-97e49232e302-workload-socket\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs\" (UID: \"aa609b95-6fbc-4cb6-9153-97e49232e302\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs"
Oct 30 05:33:13.319865 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:13.319030    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa609b95-6fbc-4cb6-9153-97e49232e302-istio-data\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs\" (UID: \"aa609b95-6fbc-4cb6-9153-97e49232e302\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs"
Oct 30 05:33:13.319865 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:13.319248    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa609b95-6fbc-4cb6-9153-97e49232e302-credential-socket\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs\" (UID: \"aa609b95-6fbc-4cb6-9153-97e49232e302\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs"
Oct 30 05:33:13.321213 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:13.320872    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa609b95-6fbc-4cb6-9153-97e49232e302-istiod-ca-cert\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs\" (UID: \"aa609b95-6fbc-4cb6-9153-97e49232e302\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs"
Oct 30 05:33:13.360970 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:13.360444    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l6hl\" (UniqueName: \"kubernetes.io/projected/aa609b95-6fbc-4cb6-9153-97e49232e302-kube-api-access-9l6hl\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs\" (UID: \"aa609b95-6fbc-4cb6-9153-97e49232e302\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs"
Oct 30 05:33:13.363963 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:13.363716    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa609b95-6fbc-4cb6-9153-97e49232e302-istio-podinfo\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs\" (UID: \"aa609b95-6fbc-4cb6-9153-97e49232e302\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs"
Oct 30 05:33:13.401224 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:13.401148    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa609b95-6fbc-4cb6-9153-97e49232e302-istio-envoy\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs\" (UID: \"aa609b95-6fbc-4cb6-9153-97e49232e302\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs"
Oct 30 05:33:13.410928 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:13.406691    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa609b95-6fbc-4cb6-9153-97e49232e302-istio-token\") pod \"e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs\" (UID: \"aa609b95-6fbc-4cb6-9153-97e49232e302\") " pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs"
Oct 30 05:33:13.665780 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:13.665746    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs"
Oct 30 05:33:13.667797 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:13.667777    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h"
Oct 30 05:33:13.667973 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:13.667824    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h" event={"ID":"6c2a73f8-80fb-46e4-ac53-50ca8ed027d9","Type":"ContainerDied","Data":"53aac1f3b976d8cf13795b55602adca9c8edbc71009da5420ce31433332ae0c4"}
Oct 30 05:33:13.667973 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:13.667889    5898 scope.go:122] "RemoveContainer" containerID="80ffd4c0bfdb6404bbba58983a4fe831f9ca4e0c7a4e4880c200434081160f34"
Oct 30 05:33:13.731618 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:13.731592    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h"]
Oct 30 05:33:13.742271 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:13.742229    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-e15c99-openshift-default-5f6599f4d7-l9g5h"]
Oct 30 05:33:13.905565 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:13.903687    5898 patch_prober.go:29] interesting pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Oct 30 05:33:13.905565 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:13.903769    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k" podUID="4c2b56cf-6fbe-4787-86db-1692ecbb41e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Oct 30 05:33:14.031670 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:33:14.031638    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa609b95_6fbc_4cb6_9153_97e49232e302.slice/crio-763e58b745e4562cd8e7f3a022f4a9f3bc9dd038a0599fb21671fede7e1221c4 WatchSource:0}: Error finding container 763e58b745e4562cd8e7f3a022f4a9f3bc9dd038a0599fb21671fede7e1221c4: Status 404 returned error can't find the container with id 763e58b745e4562cd8e7f3a022f4a9f3bc9dd038a0599fb21671fede7e1221c4
Oct 30 05:33:14.032363 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:14.032145    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs"]
Oct 30 05:33:14.036162 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:14.036126    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:33:14.036292 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:14.036214    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:33:14.036292 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:14.036264    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:33:14.726825 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:14.726768    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs" event={"ID":"aa609b95-6fbc-4cb6-9153-97e49232e302","Type":"ContainerStarted","Data":"5d1cf8101476826b36d8e1390abad24b9543c23aafadf4445db029e31647ee55"}
Oct 30 05:33:14.727523 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:14.727487    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs" event={"ID":"aa609b95-6fbc-4cb6-9153-97e49232e302","Type":"ContainerStarted","Data":"763e58b745e4562cd8e7f3a022f4a9f3bc9dd038a0599fb21671fede7e1221c4"}
Oct 30 05:33:14.886393 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:14.886357    5898 patch_prober.go:29] interesting pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Oct 30 05:33:14.886640 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:14.886612    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k" podUID="4c2b56cf-6fbe-4787-86db-1692ecbb41e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Oct 30 05:33:14.938292 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:14.938266    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c2a73f8-80fb-46e4-ac53-50ca8ed027d9" path="/var/lib/kubelet/pods/6c2a73f8-80fb-46e4-ac53-50ca8ed027d9/volumes"
Oct 30 05:33:15.327799 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:15.327761    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:15.666306 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:15.666222    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs"
Oct 30 05:33:15.667674 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:15.667653    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Oct 30 05:33:15.667779 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:15.667699    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs" podUID="aa609b95-6fbc-4cb6-9153-97e49232e302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Oct 30 05:33:15.896924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:15.896116    5898 patch_prober.go:29] interesting pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Oct 30 05:33:15.896924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:15.896180    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k" podUID="4c2b56cf-6fbe-4787-86db-1692ecbb41e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Oct 30 05:33:16.452235 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:16.452199    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2"
Oct 30 05:33:16.457246 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:16.457221    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb8ee7a6-2a74-411a-8985-c2422f295292-istio-token\" (UniqueName: \"kubernetes.io/projected/cb8ee7a6-2a74-411a-8985-c2422f295292-istio-token\") pod \"cb8ee7a6-2a74-411a-8985-c2422f295292\" (UID: \"cb8ee7a6-2a74-411a-8985-c2422f295292\") "
Oct 30 05:33:16.457380 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:16.457272    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb8ee7a6-2a74-411a-8985-c2422f295292-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb8ee7a6-2a74-411a-8985-c2422f295292-workload-certs\") pod \"cb8ee7a6-2a74-411a-8985-c2422f295292\" (UID: \"cb8ee7a6-2a74-411a-8985-c2422f295292\") "
Oct 30 05:33:16.457380 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:16.457297    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb8ee7a6-2a74-411a-8985-c2422f295292-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb8ee7a6-2a74-411a-8985-c2422f295292-istiod-ca-cert\") pod \"cb8ee7a6-2a74-411a-8985-c2422f295292\" (UID: \"cb8ee7a6-2a74-411a-8985-c2422f295292\") "
Oct 30 05:33:16.457380 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:16.457365    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb8ee7a6-2a74-411a-8985-c2422f295292-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb8ee7a6-2a74-411a-8985-c2422f295292-workload-socket\") pod \"cb8ee7a6-2a74-411a-8985-c2422f295292\" (UID: \"cb8ee7a6-2a74-411a-8985-c2422f295292\") "
Oct 30 05:33:16.457496 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:16.457478    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb8ee7a6-2a74-411a-8985-c2422f295292-kube-api-access-fnpcd\" (UniqueName: \"kubernetes.io/projected/cb8ee7a6-2a74-411a-8985-c2422f295292-kube-api-access-fnpcd\") pod \"cb8ee7a6-2a74-411a-8985-c2422f295292\" (UID: \"cb8ee7a6-2a74-411a-8985-c2422f295292\") "
Oct 30 05:33:16.457532 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:16.457500    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb8ee7a6-2a74-411a-8985-c2422f295292-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb8ee7a6-2a74-411a-8985-c2422f295292-istio-data\") pod \"cb8ee7a6-2a74-411a-8985-c2422f295292\" (UID: \"cb8ee7a6-2a74-411a-8985-c2422f295292\") "
Oct 30 05:33:16.457571 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:16.457543    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb8ee7a6-2a74-411a-8985-c2422f295292-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb8ee7a6-2a74-411a-8985-c2422f295292-credential-socket\") pod \"cb8ee7a6-2a74-411a-8985-c2422f295292\" (UID: \"cb8ee7a6-2a74-411a-8985-c2422f295292\") "
Oct 30 05:33:16.457609 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:16.457574    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb8ee7a6-2a74-411a-8985-c2422f295292-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb8ee7a6-2a74-411a-8985-c2422f295292-istio-podinfo\") pod \"cb8ee7a6-2a74-411a-8985-c2422f295292\" (UID: \"cb8ee7a6-2a74-411a-8985-c2422f295292\") "
Oct 30 05:33:16.457685 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:16.457645    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb8ee7a6-2a74-411a-8985-c2422f295292-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb8ee7a6-2a74-411a-8985-c2422f295292-istio-envoy\") pod \"cb8ee7a6-2a74-411a-8985-c2422f295292\" (UID: \"cb8ee7a6-2a74-411a-8985-c2422f295292\") "
Oct 30 05:33:16.458635 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:16.458603    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb8ee7a6-2a74-411a-8985-c2422f295292-workload-socket" pod "cb8ee7a6-2a74-411a-8985-c2422f295292" (UID: "cb8ee7a6-2a74-411a-8985-c2422f295292"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:33:16.458814 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:16.458784    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb8ee7a6-2a74-411a-8985-c2422f295292-workload-certs" pod "cb8ee7a6-2a74-411a-8985-c2422f295292" (UID: "cb8ee7a6-2a74-411a-8985-c2422f295292"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:33:16.458949 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:16.458921    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb8ee7a6-2a74-411a-8985-c2422f295292-credential-socket" pod "cb8ee7a6-2a74-411a-8985-c2422f295292" (UID: "cb8ee7a6-2a74-411a-8985-c2422f295292"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:33:16.459103 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:16.458830    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb8ee7a6-2a74-411a-8985-c2422f295292-istiod-ca-cert" pod "cb8ee7a6-2a74-411a-8985-c2422f295292" (UID: "cb8ee7a6-2a74-411a-8985-c2422f295292"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:33:16.459103 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:16.459009    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb8ee7a6-2a74-411a-8985-c2422f295292-istio-data" pod "cb8ee7a6-2a74-411a-8985-c2422f295292" (UID: "cb8ee7a6-2a74-411a-8985-c2422f295292"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:33:16.460890 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:16.460863    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb8ee7a6-2a74-411a-8985-c2422f295292-istio-envoy" pod "cb8ee7a6-2a74-411a-8985-c2422f295292" (UID: "cb8ee7a6-2a74-411a-8985-c2422f295292"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:33:16.461116 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:16.461095    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb8ee7a6-2a74-411a-8985-c2422f295292-kube-api-access-fnpcd" pod "cb8ee7a6-2a74-411a-8985-c2422f295292" (UID: "cb8ee7a6-2a74-411a-8985-c2422f295292"). InnerVolumeSpecName "kube-api-access-fnpcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:33:16.461351 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:16.461311    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb8ee7a6-2a74-411a-8985-c2422f295292-istio-token" pod "cb8ee7a6-2a74-411a-8985-c2422f295292" (UID: "cb8ee7a6-2a74-411a-8985-c2422f295292"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:33:16.464403 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:16.464379    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb8ee7a6-2a74-411a-8985-c2422f295292-istio-podinfo" pod "cb8ee7a6-2a74-411a-8985-c2422f295292" (UID: "cb8ee7a6-2a74-411a-8985-c2422f295292"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:33:16.545827 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:16.545760    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs" podStartSLOduration=171.545741519 podStartE2EDuration="2m51.545741519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:33:14.783699222 +0000 UTC m=+2924.007519150" watchObservedRunningTime="2026-10-30 05:33:16.545741519 +0000 UTC m=+2925.769561423"
Oct 30 05:33:16.560659 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:16.559543    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnpcd\" (UniqueName: \"kubernetes.io/projected/cb8ee7a6-2a74-411a-8985-c2422f295292-kube-api-access-fnpcd\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:33:16.560659 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:16.559585    5898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb8ee7a6-2a74-411a-8985-c2422f295292-istio-data\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:33:16.560659 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:16.559602    5898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb8ee7a6-2a74-411a-8985-c2422f295292-credential-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:33:16.560659 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:16.559617    5898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb8ee7a6-2a74-411a-8985-c2422f295292-istio-podinfo\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:33:16.560659 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:16.559631    5898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb8ee7a6-2a74-411a-8985-c2422f295292-istio-envoy\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:33:16.560659 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:16.559643    5898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb8ee7a6-2a74-411a-8985-c2422f295292-istio-token\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:33:16.560659 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:16.559655    5898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb8ee7a6-2a74-411a-8985-c2422f295292-workload-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:33:16.560659 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:16.559670    5898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb8ee7a6-2a74-411a-8985-c2422f295292-istiod-ca-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:33:16.560659 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:16.559683    5898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb8ee7a6-2a74-411a-8985-c2422f295292-workload-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:33:16.666849 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:16.666815    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Oct 30 05:33:16.667047 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:16.666868    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs" podUID="aa609b95-6fbc-4cb6-9153-97e49232e302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Oct 30 05:33:16.748343 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:16.748249    5898 generic.go:356] "Generic (PLEG): container finished" podID="cb8ee7a6-2a74-411a-8985-c2422f295292" containerID="dab4512284b5d8df32e9e348f6d2614ee855a106c6a3ef4c763b396cc3c1a072" exitCode=0
Oct 30 05:33:16.748343 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:16.748314    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2"
Oct 30 05:33:16.748594 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:16.748473    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2" event={"ID":"cb8ee7a6-2a74-411a-8985-c2422f295292","Type":"ContainerDied","Data":"dab4512284b5d8df32e9e348f6d2614ee855a106c6a3ef4c763b396cc3c1a072"}
Oct 30 05:33:16.748594 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:16.748513    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2" event={"ID":"cb8ee7a6-2a74-411a-8985-c2422f295292","Type":"ContainerDied","Data":"d902d7b03df409e39fa8b58ce6512521e26485e62ada7e15bcf44f8d45a0131e"}
Oct 30 05:33:16.748594 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:16.748534    5898 scope.go:122] "RemoveContainer" containerID="dab4512284b5d8df32e9e348f6d2614ee855a106c6a3ef4c763b396cc3c1a072"
Oct 30 05:33:16.765156 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:16.765055    5898 scope.go:122] "RemoveContainer" containerID="dab4512284b5d8df32e9e348f6d2614ee855a106c6a3ef4c763b396cc3c1a072"
Oct 30 05:33:16.766464 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:33:16.766419    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dab4512284b5d8df32e9e348f6d2614ee855a106c6a3ef4c763b396cc3c1a072\": container with ID starting with dab4512284b5d8df32e9e348f6d2614ee855a106c6a3ef4c763b396cc3c1a072 not found: ID does not exist" containerID="dab4512284b5d8df32e9e348f6d2614ee855a106c6a3ef4c763b396cc3c1a072"
Oct 30 05:33:16.766464 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:16.766450    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dab4512284b5d8df32e9e348f6d2614ee855a106c6a3ef4c763b396cc3c1a072"} err="failed to get container status \"dab4512284b5d8df32e9e348f6d2614ee855a106c6a3ef4c763b396cc3c1a072\": rpc error: code = NotFound desc = could not find container \"dab4512284b5d8df32e9e348f6d2614ee855a106c6a3ef4c763b396cc3c1a072\": container with ID starting with dab4512284b5d8df32e9e348f6d2614ee855a106c6a3ef4c763b396cc3c1a072 not found: ID does not exist"
Oct 30 05:33:16.831645 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:16.831609    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2"]
Oct 30 05:33:16.835811 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:16.835786    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-45607d-openshift-default-5988476fcc-9drs2"]
Oct 30 05:33:16.899834 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:16.884308    5898 patch_prober.go:29] interesting pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Oct 30 05:33:16.899834 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:16.884367    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k" podUID="4c2b56cf-6fbe-4787-86db-1692ecbb41e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Oct 30 05:33:16.937733 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:16.937698    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb8ee7a6-2a74-411a-8985-c2422f295292" path="/var/lib/kubelet/pods/cb8ee7a6-2a74-411a-8985-c2422f295292/volumes"
Oct 30 05:33:17.279928 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:17.278558    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:17.666106 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:17.666072    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Oct 30 05:33:17.666284 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:17.666122    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs" podUID="aa609b95-6fbc-4cb6-9153-97e49232e302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Oct 30 05:33:17.881936 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:17.878313    5898 patch_prober.go:29] interesting pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Oct 30 05:33:17.881936 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:17.878377    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k" podUID="4c2b56cf-6fbe-4787-86db-1692ecbb41e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Oct 30 05:33:18.032344 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:18.032224    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4"]
Oct 30 05:33:18.034137 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:18.034111    5898 cpu_manager.go:414] "containerMap: removing container" podUID="cb8ee7a6-2a74-411a-8985-c2422f295292" containerName="istio-proxy"
Oct 30 05:33:18.034326 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:18.034312    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb8ee7a6-2a74-411a-8985-c2422f295292" containerName="istio-proxy"
Oct 30 05:33:18.034662 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:18.034637    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb8ee7a6-2a74-411a-8985-c2422f295292" containerName="istio-proxy"
Oct 30 05:33:18.057925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:18.053373    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4"]
Oct 30 05:33:18.057925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:18.053523    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4"
Oct 30 05:33:18.065633 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:18.059466    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:18.085002 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:18.083266    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-262c10-openshift-default-dockercfg-gq5xt\""
Oct 30 05:33:18.243924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:18.227830    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bc29954-1331-4bdf-9d6c-57e5894a5947-istio-podinfo\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4\" (UID: \"3bc29954-1331-4bdf-9d6c-57e5894a5947\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4"
Oct 30 05:33:18.243924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:18.227978    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bc29954-1331-4bdf-9d6c-57e5894a5947-istiod-ca-cert\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4\" (UID: \"3bc29954-1331-4bdf-9d6c-57e5894a5947\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4"
Oct 30 05:33:18.243924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:18.228035    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n422v\" (UniqueName: \"kubernetes.io/projected/3bc29954-1331-4bdf-9d6c-57e5894a5947-kube-api-access-n422v\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4\" (UID: \"3bc29954-1331-4bdf-9d6c-57e5894a5947\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4"
Oct 30 05:33:18.243924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:18.228070    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc29954-1331-4bdf-9d6c-57e5894a5947-credential-socket\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4\" (UID: \"3bc29954-1331-4bdf-9d6c-57e5894a5947\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4"
Oct 30 05:33:18.243924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:18.228127    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bc29954-1331-4bdf-9d6c-57e5894a5947-istio-envoy\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4\" (UID: \"3bc29954-1331-4bdf-9d6c-57e5894a5947\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4"
Oct 30 05:33:18.243924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:18.228182    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bc29954-1331-4bdf-9d6c-57e5894a5947-istio-data\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4\" (UID: \"3bc29954-1331-4bdf-9d6c-57e5894a5947\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4"
Oct 30 05:33:18.243924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:18.228208    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bc29954-1331-4bdf-9d6c-57e5894a5947-istio-token\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4\" (UID: \"3bc29954-1331-4bdf-9d6c-57e5894a5947\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4"
Oct 30 05:33:18.243924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:18.228234    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bc29954-1331-4bdf-9d6c-57e5894a5947-workload-certs\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4\" (UID: \"3bc29954-1331-4bdf-9d6c-57e5894a5947\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4"
Oct 30 05:33:18.243924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:18.228300    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc29954-1331-4bdf-9d6c-57e5894a5947-workload-socket\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4\" (UID: \"3bc29954-1331-4bdf-9d6c-57e5894a5947\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4"
Oct 30 05:33:18.243924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:18.242463    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:18.330962 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:18.330890    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bc29954-1331-4bdf-9d6c-57e5894a5947-istio-envoy\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4\" (UID: \"3bc29954-1331-4bdf-9d6c-57e5894a5947\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4"
Oct 30 05:33:18.331189 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:18.330993    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bc29954-1331-4bdf-9d6c-57e5894a5947-istio-data\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4\" (UID: \"3bc29954-1331-4bdf-9d6c-57e5894a5947\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4"
Oct 30 05:33:18.331189 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:18.331016    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bc29954-1331-4bdf-9d6c-57e5894a5947-istio-token\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4\" (UID: \"3bc29954-1331-4bdf-9d6c-57e5894a5947\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4"
Oct 30 05:33:18.331189 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:18.331038    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bc29954-1331-4bdf-9d6c-57e5894a5947-workload-certs\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4\" (UID: \"3bc29954-1331-4bdf-9d6c-57e5894a5947\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4"
Oct 30 05:33:18.331189 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:18.331102    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc29954-1331-4bdf-9d6c-57e5894a5947-workload-socket\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4\" (UID: \"3bc29954-1331-4bdf-9d6c-57e5894a5947\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4"
Oct 30 05:33:18.331406 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:18.331306    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bc29954-1331-4bdf-9d6c-57e5894a5947-istio-podinfo\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4\" (UID: \"3bc29954-1331-4bdf-9d6c-57e5894a5947\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4"
Oct 30 05:33:18.331467 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:18.331435    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bc29954-1331-4bdf-9d6c-57e5894a5947-istiod-ca-cert\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4\" (UID: \"3bc29954-1331-4bdf-9d6c-57e5894a5947\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4"
Oct 30 05:33:18.331521 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:18.331491    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n422v\" (UniqueName: \"kubernetes.io/projected/3bc29954-1331-4bdf-9d6c-57e5894a5947-kube-api-access-n422v\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4\" (UID: \"3bc29954-1331-4bdf-9d6c-57e5894a5947\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4"
Oct 30 05:33:18.331588 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:18.331526    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc29954-1331-4bdf-9d6c-57e5894a5947-credential-socket\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4\" (UID: \"3bc29954-1331-4bdf-9d6c-57e5894a5947\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4"
Oct 30 05:33:18.332099 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:18.332077    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bc29954-1331-4bdf-9d6c-57e5894a5947-workload-certs\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4\" (UID: \"3bc29954-1331-4bdf-9d6c-57e5894a5947\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4"
Oct 30 05:33:18.332240 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:18.332063    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc29954-1331-4bdf-9d6c-57e5894a5947-credential-socket\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4\" (UID: \"3bc29954-1331-4bdf-9d6c-57e5894a5947\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4"
Oct 30 05:33:18.332355 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:18.332315    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc29954-1331-4bdf-9d6c-57e5894a5947-workload-socket\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4\" (UID: \"3bc29954-1331-4bdf-9d6c-57e5894a5947\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4"
Oct 30 05:33:18.332919 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:18.332883    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bc29954-1331-4bdf-9d6c-57e5894a5947-istiod-ca-cert\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4\" (UID: \"3bc29954-1331-4bdf-9d6c-57e5894a5947\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4"
Oct 30 05:33:18.333027 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:18.332887    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bc29954-1331-4bdf-9d6c-57e5894a5947-istio-data\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4\" (UID: \"3bc29954-1331-4bdf-9d6c-57e5894a5947\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4"
Oct 30 05:33:18.335781 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:18.335760    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bc29954-1331-4bdf-9d6c-57e5894a5947-istio-envoy\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4\" (UID: \"3bc29954-1331-4bdf-9d6c-57e5894a5947\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4"
Oct 30 05:33:18.336165 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:18.336146    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bc29954-1331-4bdf-9d6c-57e5894a5947-istio-podinfo\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4\" (UID: \"3bc29954-1331-4bdf-9d6c-57e5894a5947\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4"
Oct 30 05:33:18.355417 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:18.354935    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n422v\" (UniqueName: \"kubernetes.io/projected/3bc29954-1331-4bdf-9d6c-57e5894a5947-kube-api-access-n422v\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4\" (UID: \"3bc29954-1331-4bdf-9d6c-57e5894a5947\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4"
Oct 30 05:33:18.355417 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:18.355096    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bc29954-1331-4bdf-9d6c-57e5894a5947-istio-token\") pod \"e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4\" (UID: \"3bc29954-1331-4bdf-9d6c-57e5894a5947\") " pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4"
Oct 30 05:33:18.456934 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:18.453630    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4"
Oct 30 05:33:18.694602 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:18.694513    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Oct 30 05:33:18.694873 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:18.694844    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs" podUID="aa609b95-6fbc-4cb6-9153-97e49232e302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Oct 30 05:33:18.908353 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:18.905397    5898 patch_prober.go:29] interesting pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Oct 30 05:33:18.908353 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:18.905455    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k" podUID="4c2b56cf-6fbe-4787-86db-1692ecbb41e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Oct 30 05:33:19.138024 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:19.137995    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4"]
Oct 30 05:33:19.140575 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:33:19.140546    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bc29954_1331_4bdf_9d6c_57e5894a5947.slice/crio-fd1a63addc6502ad91f4197ff8d90fba711885265ddd20d0f1abf1737d424cab WatchSource:0}: Error finding container fd1a63addc6502ad91f4197ff8d90fba711885265ddd20d0f1abf1737d424cab: Status 404 returned error can't find the container with id fd1a63addc6502ad91f4197ff8d90fba711885265ddd20d0f1abf1737d424cab
Oct 30 05:33:19.143948 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:19.143895    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:33:19.144072 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:19.144035    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:33:19.144143 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:19.144088    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:33:19.393979 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:19.391427    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:19.666935 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:19.666827    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Oct 30 05:33:19.666935 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:19.666876    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs" podUID="aa609b95-6fbc-4cb6-9153-97e49232e302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Oct 30 05:33:19.827387 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:19.824443    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4" event={"ID":"3bc29954-1331-4bdf-9d6c-57e5894a5947","Type":"ContainerStarted","Data":"40d770fe1a0221210344d1f60c830b9d991dbea16f042644d676602cfd14bf67"}
Oct 30 05:33:19.827387 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:19.824499    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4" event={"ID":"3bc29954-1331-4bdf-9d6c-57e5894a5947","Type":"ContainerStarted","Data":"fd1a63addc6502ad91f4197ff8d90fba711885265ddd20d0f1abf1737d424cab"}
Oct 30 05:33:19.920662 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:19.918314    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k"
Oct 30 05:33:19.920855 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:19.920840    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k"
Oct 30 05:33:19.935947 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:19.924972    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4" podStartSLOduration=2.924952431 podStartE2EDuration="2.924952431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:33:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:33:19.905502013 +0000 UTC m=+2929.129321913" watchObservedRunningTime="2026-10-30 05:33:19.924952431 +0000 UTC m=+2929.148772334"
Oct 30 05:33:19.940420 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:19.940066    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k"
Oct 30 05:33:20.454591 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:20.454557    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4"
Oct 30 05:33:20.456448 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:20.456416    5898 patch_prober.go:29] interesting pod/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 05:33:20.456572 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:20.456473    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4" podUID="3bc29954-1331-4bdf-9d6c-57e5894a5947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 05:33:20.666893 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:20.666852    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Oct 30 05:33:20.666893 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:20.666916    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs" podUID="aa609b95-6fbc-4cb6-9153-97e49232e302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Oct 30 05:33:21.372762 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:21.372659    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:21.464923 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:21.458943    5898 patch_prober.go:29] interesting pod/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 05:33:21.464923 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:21.459012    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4" podUID="3bc29954-1331-4bdf-9d6c-57e5894a5947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 05:33:21.671934 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:21.671844    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs"
Oct 30 05:33:21.672294 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:21.672277    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs"
Oct 30 05:33:21.673054 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:21.673032    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs"
Oct 30 05:33:22.165837 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:22.165796    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:22.315535 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:22.315481    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:22.454963 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:22.454865    5898 patch_prober.go:29] interesting pod/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 05:33:22.455141 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:22.454959    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4" podUID="3bc29954-1331-4bdf-9d6c-57e5894a5947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 05:33:23.454696 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:23.454659    5898 patch_prober.go:29] interesting pod/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 05:33:23.455137 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:23.454711    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4" podUID="3bc29954-1331-4bdf-9d6c-57e5894a5947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 05:33:23.933765 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:23.933731    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:24.454290 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:24.454253    5898 patch_prober.go:29] interesting pod/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 05:33:24.454502 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:24.454312    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4" podUID="3bc29954-1331-4bdf-9d6c-57e5894a5947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 05:33:24.734515 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:24.734415    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-02cffa-858d55bbb-hxc8w"
Oct 30 05:33:25.069440 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:25.069353    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:25.454731 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:25.454695    5898 patch_prober.go:29] interesting pod/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 05:33:25.454958 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:25.454760    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4" podUID="3bc29954-1331-4bdf-9d6c-57e5894a5947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 05:33:25.620851 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:25.620802    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:26.376220 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:26.376182    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:26.615336 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:26.615303    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4"
Oct 30 05:33:26.615989 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:26.615944    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4"
Oct 30 05:33:26.616830 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:26.616805    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4"
Oct 30 05:33:26.829099 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:33:26.829052    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:33:27.011153 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:27.011120    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247" podUID="1d865d3e-a6e9-407e-844e-cc75fc7f1354" containerName="main" containerID="cri-o://184b8e65fb984b00278e69cb1a81946da12ac9ff7d0733cfb1bcc254412d30c8" gracePeriod=45
Oct 30 05:33:27.265099 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:27.249068    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:27.316767 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:27.316737    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247"
Oct 30 05:33:27.339938 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:27.339869    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d865d3e-a6e9-407e-844e-cc75fc7f1354-tls-certs\" (UniqueName: \"kubernetes.io/secret/1d865d3e-a6e9-407e-844e-cc75fc7f1354-tls-certs\") pod \"1d865d3e-a6e9-407e-844e-cc75fc7f1354\" (UID: \"1d865d3e-a6e9-407e-844e-cc75fc7f1354\") "
Oct 30 05:33:27.340116 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:27.340050    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d865d3e-a6e9-407e-844e-cc75fc7f1354-kube-api-access-nhp8f\" (UniqueName: \"kubernetes.io/projected/1d865d3e-a6e9-407e-844e-cc75fc7f1354-kube-api-access-nhp8f\") pod \"1d865d3e-a6e9-407e-844e-cc75fc7f1354\" (UID: \"1d865d3e-a6e9-407e-844e-cc75fc7f1354\") "
Oct 30 05:33:27.340116 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:27.340091    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d865d3e-a6e9-407e-844e-cc75fc7f1354-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d865d3e-a6e9-407e-844e-cc75fc7f1354-dshm\") pod \"1d865d3e-a6e9-407e-844e-cc75fc7f1354\" (UID: \"1d865d3e-a6e9-407e-844e-cc75fc7f1354\") "
Oct 30 05:33:27.340222 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:27.340126    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d865d3e-a6e9-407e-844e-cc75fc7f1354-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d865d3e-a6e9-407e-844e-cc75fc7f1354-model-cache\") pod \"1d865d3e-a6e9-407e-844e-cc75fc7f1354\" (UID: \"1d865d3e-a6e9-407e-844e-cc75fc7f1354\") "
Oct 30 05:33:27.340284 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:27.340222    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d865d3e-a6e9-407e-844e-cc75fc7f1354-home\" (UniqueName: \"kubernetes.io/empty-dir/1d865d3e-a6e9-407e-844e-cc75fc7f1354-home\") pod \"1d865d3e-a6e9-407e-844e-cc75fc7f1354\" (UID: \"1d865d3e-a6e9-407e-844e-cc75fc7f1354\") "
Oct 30 05:33:27.340408 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:27.340389    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d865d3e-a6e9-407e-844e-cc75fc7f1354-model-cache" pod "1d865d3e-a6e9-407e-844e-cc75fc7f1354" (UID: "1d865d3e-a6e9-407e-844e-cc75fc7f1354"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:33:27.340519 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:27.340495    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d865d3e-a6e9-407e-844e-cc75fc7f1354-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d865d3e-a6e9-407e-844e-cc75fc7f1354-tmp-dir\") pod \"1d865d3e-a6e9-407e-844e-cc75fc7f1354\" (UID: \"1d865d3e-a6e9-407e-844e-cc75fc7f1354\") "
Oct 30 05:33:27.340714 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:27.340676    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d865d3e-a6e9-407e-844e-cc75fc7f1354-home" pod "1d865d3e-a6e9-407e-844e-cc75fc7f1354" (UID: "1d865d3e-a6e9-407e-844e-cc75fc7f1354"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:33:27.340982 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:27.340960    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d865d3e-a6e9-407e-844e-cc75fc7f1354-tmp-dir" pod "1d865d3e-a6e9-407e-844e-cc75fc7f1354" (UID: "1d865d3e-a6e9-407e-844e-cc75fc7f1354"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:33:27.341232 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:27.341216    5898 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d865d3e-a6e9-407e-844e-cc75fc7f1354-model-cache\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:33:27.341315 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:27.341235    5898 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d865d3e-a6e9-407e-844e-cc75fc7f1354-home\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:33:27.341315 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:27.341244    5898 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d865d3e-a6e9-407e-844e-cc75fc7f1354-tmp-dir\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:33:27.342688 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:27.342657    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d865d3e-a6e9-407e-844e-cc75fc7f1354-kube-api-access-nhp8f" pod "1d865d3e-a6e9-407e-844e-cc75fc7f1354" (UID: "1d865d3e-a6e9-407e-844e-cc75fc7f1354"). InnerVolumeSpecName "kube-api-access-nhp8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:33:27.342774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:27.342665    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d865d3e-a6e9-407e-844e-cc75fc7f1354-dshm" pod "1d865d3e-a6e9-407e-844e-cc75fc7f1354" (UID: "1d865d3e-a6e9-407e-844e-cc75fc7f1354"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:33:27.342774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:27.342666    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d865d3e-a6e9-407e-844e-cc75fc7f1354-tls-certs" pod "1d865d3e-a6e9-407e-844e-cc75fc7f1354" (UID: "1d865d3e-a6e9-407e-844e-cc75fc7f1354"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:33:27.443426 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:27.442965    5898 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d865d3e-a6e9-407e-844e-cc75fc7f1354-tls-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:33:27.443985 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:27.443949    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhp8f\" (UniqueName: \"kubernetes.io/projected/1d865d3e-a6e9-407e-844e-cc75fc7f1354-kube-api-access-nhp8f\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:33:27.444107 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:27.443992    5898 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d865d3e-a6e9-407e-844e-cc75fc7f1354-dshm\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:33:27.561937 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:27.561861    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:27.988622 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:27.986134    5898 generic.go:356] "Generic (PLEG): container finished" podID="1d865d3e-a6e9-407e-844e-cc75fc7f1354" containerID="184b8e65fb984b00278e69cb1a81946da12ac9ff7d0733cfb1bcc254412d30c8" exitCode=0
Oct 30 05:33:27.988622 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:27.987414    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247"
Oct 30 05:33:27.993388 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:27.993351    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247" event={"ID":"1d865d3e-a6e9-407e-844e-cc75fc7f1354","Type":"ContainerDied","Data":"184b8e65fb984b00278e69cb1a81946da12ac9ff7d0733cfb1bcc254412d30c8"}
Oct 30 05:33:27.993805 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:27.993779    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247" event={"ID":"1d865d3e-a6e9-407e-844e-cc75fc7f1354","Type":"ContainerDied","Data":"4f0b238e0e5670439e0b22c2caa00ac0a5759645260914360adb395ea1fd376c"}
Oct 30 05:33:27.993978 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:27.993962    5898 scope.go:122] "RemoveContainer" containerID="184b8e65fb984b00278e69cb1a81946da12ac9ff7d0733cfb1bcc254412d30c8"
Oct 30 05:33:28.020653 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:28.020475    5898 scope.go:122] "RemoveContainer" containerID="184b8e65fb984b00278e69cb1a81946da12ac9ff7d0733cfb1bcc254412d30c8"
Oct 30 05:33:28.020891 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:33:28.020864    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"184b8e65fb984b00278e69cb1a81946da12ac9ff7d0733cfb1bcc254412d30c8\": container with ID starting with 184b8e65fb984b00278e69cb1a81946da12ac9ff7d0733cfb1bcc254412d30c8 not found: ID does not exist" containerID="184b8e65fb984b00278e69cb1a81946da12ac9ff7d0733cfb1bcc254412d30c8"
Oct 30 05:33:28.021026 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:28.020937    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"184b8e65fb984b00278e69cb1a81946da12ac9ff7d0733cfb1bcc254412d30c8"} err="failed to get container status \"184b8e65fb984b00278e69cb1a81946da12ac9ff7d0733cfb1bcc254412d30c8\": rpc error: code = NotFound desc = could not find container \"184b8e65fb984b00278e69cb1a81946da12ac9ff7d0733cfb1bcc254412d30c8\": container with ID starting with 184b8e65fb984b00278e69cb1a81946da12ac9ff7d0733cfb1bcc254412d30c8 not found: ID does not exist"
Oct 30 05:33:28.063768 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:28.063677    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247"]
Oct 30 05:33:28.068420 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:28.068394    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-889ca9/premium-w3-889ca9-kserve-57dd9bb5fc-ql247"]
Oct 30 05:33:28.100608 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:28.100573    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:28.892786 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:28.892748    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d865d3e-a6e9-407e-844e-cc75fc7f1354" path="/var/lib/kubelet/pods/1d865d3e-a6e9-407e-844e-cc75fc7f1354/volumes"
Oct 30 05:33:29.855790 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:29.855760    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:31.360284 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:31.360247    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:31.535522 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:31.535485    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:31.807425 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:31.807343    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:32.479512 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:32.478978    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:34.180529 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:34.180499    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:34.331383 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:34.331331    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:34.441271 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:34.441197    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:34.599367 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:34.599329    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:35.002497 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:35.002445    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:36.456012 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:36.455967    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:37.225228 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:37.225184    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:37.867214 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:37.867155    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:38.390603 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:38.390574    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:38.561430 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:38.561395    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:39.215984 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:39.215946    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:40.646682 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:40.646644    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:33:41.382552 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:41.382517    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:42.509935 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:42.509868    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:42.642573 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:42.642532    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:43.343762 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:43.343732    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:44.000979 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:44.000949    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:44.448090 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:44.448058    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:44.615527 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:44.615493    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:46.286536 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:46.286487    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:46.396483 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:46.396449    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:46.522992 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:46.522961    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:48.702481 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:48.702435    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:49.856334 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:49.856302    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:50.021412 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:50.021289    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:50.821176 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:50.821139    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:50.986980 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:50.986928    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:52.877781 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:52.877745    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:54.892275 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:54.892244    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:56.723712 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:56.723676    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:56.820087 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:33:56.820054    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:33:56.892583 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:56.892548    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b5b27e-6fc7474f79-lk2jv"]
Oct 30 05:33:56.998183 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:56.998048    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:57.106134 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:57.106099    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:58.377855 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:58.377825    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:33:59.453566 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:33:59.453531    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:01.817280 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:01.817250    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:01.901648 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:01.900971    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5b27e-6fc7474f79-lk2jv" podUID="4e8672db-7de8-4aee-8b3b-46ccdc97a211" containerName="payload-pre-processing" containerID="cri-o://728f6c799e487753ff890dad10f402f2e61eb26a10d5d0a11184a6447ec44fb6" gracePeriod=25
Oct 30 05:34:02.199740 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:02.199714    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5b27e-6fc7474f79-lk2jv"
Oct 30 05:34:02.301578 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:02.301541    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e8672db-7de8-4aee-8b3b-46ccdc97a211-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e8672db-7de8-4aee-8b3b-46ccdc97a211-plugins-config-volume\") pod \"4e8672db-7de8-4aee-8b3b-46ccdc97a211\" (UID: \"4e8672db-7de8-4aee-8b3b-46ccdc97a211\") "
Oct 30 05:34:02.301756 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:02.301623    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e8672db-7de8-4aee-8b3b-46ccdc97a211-kube-api-access-bkgn5\" (UniqueName: \"kubernetes.io/projected/4e8672db-7de8-4aee-8b3b-46ccdc97a211-kube-api-access-bkgn5\") pod \"4e8672db-7de8-4aee-8b3b-46ccdc97a211\" (UID: \"4e8672db-7de8-4aee-8b3b-46ccdc97a211\") "
Oct 30 05:34:02.302090 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:02.302063    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e8672db-7de8-4aee-8b3b-46ccdc97a211-plugins-config-volume" pod "4e8672db-7de8-4aee-8b3b-46ccdc97a211" (UID: "4e8672db-7de8-4aee-8b3b-46ccdc97a211"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:34:02.302601 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:02.302575    5898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e8672db-7de8-4aee-8b3b-46ccdc97a211-plugins-config-volume\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:34:02.303984 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:02.303959    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e8672db-7de8-4aee-8b3b-46ccdc97a211-kube-api-access-bkgn5" pod "4e8672db-7de8-4aee-8b3b-46ccdc97a211" (UID: "4e8672db-7de8-4aee-8b3b-46ccdc97a211"). InnerVolumeSpecName "kube-api-access-bkgn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:34:02.404320 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:02.404238    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkgn5\" (UniqueName: \"kubernetes.io/projected/4e8672db-7de8-4aee-8b3b-46ccdc97a211-kube-api-access-bkgn5\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:34:02.634973 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:02.634938    5898 generic.go:356] "Generic (PLEG): container finished" podID="4e8672db-7de8-4aee-8b3b-46ccdc97a211" containerID="728f6c799e487753ff890dad10f402f2e61eb26a10d5d0a11184a6447ec44fb6" exitCode=0
Oct 30 05:34:02.635168 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:02.634979    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5b27e-6fc7474f79-lk2jv" event={"ID":"4e8672db-7de8-4aee-8b3b-46ccdc97a211","Type":"ContainerDied","Data":"728f6c799e487753ff890dad10f402f2e61eb26a10d5d0a11184a6447ec44fb6"}
Oct 30 05:34:02.635168 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:02.635001    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5b27e-6fc7474f79-lk2jv"
Oct 30 05:34:02.635168 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:02.635093    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b5b27e-6fc7474f79-lk2jv" event={"ID":"4e8672db-7de8-4aee-8b3b-46ccdc97a211","Type":"ContainerDied","Data":"24d39e62b1d8963efc240796651879ba335e73592cb1675ebd36ca0368d221e0"}
Oct 30 05:34:02.635168 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:02.635126    5898 scope.go:122] "RemoveContainer" containerID="728f6c799e487753ff890dad10f402f2e61eb26a10d5d0a11184a6447ec44fb6"
Oct 30 05:34:02.654116 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:02.654087    5898 scope.go:122] "RemoveContainer" containerID="728f6c799e487753ff890dad10f402f2e61eb26a10d5d0a11184a6447ec44fb6"
Oct 30 05:34:02.654682 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:34:02.654618    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"728f6c799e487753ff890dad10f402f2e61eb26a10d5d0a11184a6447ec44fb6\": container with ID starting with 728f6c799e487753ff890dad10f402f2e61eb26a10d5d0a11184a6447ec44fb6 not found: ID does not exist" containerID="728f6c799e487753ff890dad10f402f2e61eb26a10d5d0a11184a6447ec44fb6"
Oct 30 05:34:02.654682 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:02.654657    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"728f6c799e487753ff890dad10f402f2e61eb26a10d5d0a11184a6447ec44fb6"} err="failed to get container status \"728f6c799e487753ff890dad10f402f2e61eb26a10d5d0a11184a6447ec44fb6\": rpc error: code = NotFound desc = could not find container \"728f6c799e487753ff890dad10f402f2e61eb26a10d5d0a11184a6447ec44fb6\": container with ID starting with 728f6c799e487753ff890dad10f402f2e61eb26a10d5d0a11184a6447ec44fb6 not found: ID does not exist"
Oct 30 05:34:02.689307 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:02.689273    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b5b27e-6fc7474f79-lk2jv"]
Oct 30 05:34:02.703945 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:02.703890    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b5b27e-6fc7474f79-lk2jv"]
Oct 30 05:34:02.893292 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:02.893264    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e8672db-7de8-4aee-8b3b-46ccdc97a211" path="/var/lib/kubelet/pods/4e8672db-7de8-4aee-8b3b-46ccdc97a211/volumes"
Oct 30 05:34:03.785805 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:03.785773    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:04.350401 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:04.350370    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:05.541319 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:05.541286    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:07.755738 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:07.755702    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:09.967869 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:09.967811    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:10.640814 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:10.640761    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:10.761035 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:10.761004    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:10.930813 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:10.930579    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:11.631605 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:11.631576    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:11.760567 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:11.760536    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:12.193149 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:12.193115    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:14.147529 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:14.147495    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:16.075828 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:16.075794    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:18.497714 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:18.497678    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:20.243313 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:20.243281    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:20.353648 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:20.353615    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:20.460226 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:20.460191    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:22.416893 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:22.416859    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:24.286977 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:24.286937    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:26.146331 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:26.146281    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:26.821647 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:34:26.821610    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:34:28.176615 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:28.176539    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-a914e9-777dc7878c-j4drn"]
Oct 30 05:34:28.177084 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:28.176832    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-a914e9-777dc7878c-j4drn" podUID="132139a5-1000-4abb-8964-d721b3a1bc37" containerName="maas-api" containerID="cri-o://cdc8942d9db2c701ccb2f4c339c37324bbc7949f0e4e8af925bf7b7b92af8ae9" gracePeriod=30
Oct 30 05:34:28.244616 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:28.244375    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:28.919894 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:28.919853    5898 generic.go:356] "Generic (PLEG): container finished" podID="132139a5-1000-4abb-8964-d721b3a1bc37" containerID="cdc8942d9db2c701ccb2f4c339c37324bbc7949f0e4e8af925bf7b7b92af8ae9" exitCode=0
Oct 30 05:34:28.968770 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:28.967354    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-a914e9-777dc7878c-j4drn" event={"ID":"132139a5-1000-4abb-8964-d721b3a1bc37","Type":"ContainerDied","Data":"cdc8942d9db2c701ccb2f4c339c37324bbc7949f0e4e8af925bf7b7b92af8ae9"}
Oct 30 05:34:28.987845 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:28.987807    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:28.996160 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:28.996130    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-a914e9-777dc7878c-j4drn"
Oct 30 05:34:29.099522 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:29.097403    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/132139a5-1000-4abb-8964-d721b3a1bc37-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/132139a5-1000-4abb-8964-d721b3a1bc37-openshift-service-ca\") pod \"132139a5-1000-4abb-8964-d721b3a1bc37\" (UID: \"132139a5-1000-4abb-8964-d721b3a1bc37\") "
Oct 30 05:34:29.099522 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:29.097633    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/132139a5-1000-4abb-8964-d721b3a1bc37-metrics-certs\" (UniqueName: \"kubernetes.io/secret/132139a5-1000-4abb-8964-d721b3a1bc37-metrics-certs\") pod \"132139a5-1000-4abb-8964-d721b3a1bc37\" (UID: \"132139a5-1000-4abb-8964-d721b3a1bc37\") "
Oct 30 05:34:29.099522 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:29.097800    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/132139a5-1000-4abb-8964-d721b3a1bc37-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/132139a5-1000-4abb-8964-d721b3a1bc37-maas-api-tls\") pod \"132139a5-1000-4abb-8964-d721b3a1bc37\" (UID: \"132139a5-1000-4abb-8964-d721b3a1bc37\") "
Oct 30 05:34:29.099522 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:29.097868    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/132139a5-1000-4abb-8964-d721b3a1bc37-kube-api-access-tgq9d\" (UniqueName: \"kubernetes.io/projected/132139a5-1000-4abb-8964-d721b3a1bc37-kube-api-access-tgq9d\") pod \"132139a5-1000-4abb-8964-d721b3a1bc37\" (UID: \"132139a5-1000-4abb-8964-d721b3a1bc37\") "
Oct 30 05:34:29.099522 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:29.098270    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/132139a5-1000-4abb-8964-d721b3a1bc37-openshift-service-ca" pod "132139a5-1000-4abb-8964-d721b3a1bc37" (UID: "132139a5-1000-4abb-8964-d721b3a1bc37"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:34:29.099522 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:29.098663    5898 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/132139a5-1000-4abb-8964-d721b3a1bc37-openshift-service-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:34:29.108992 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:29.108948    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/132139a5-1000-4abb-8964-d721b3a1bc37-kube-api-access-tgq9d" pod "132139a5-1000-4abb-8964-d721b3a1bc37" (UID: "132139a5-1000-4abb-8964-d721b3a1bc37"). InnerVolumeSpecName "kube-api-access-tgq9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:34:29.113383 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:29.113357    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/132139a5-1000-4abb-8964-d721b3a1bc37-maas-api-tls" pod "132139a5-1000-4abb-8964-d721b3a1bc37" (UID: "132139a5-1000-4abb-8964-d721b3a1bc37"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:34:29.115177 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:29.115139    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/132139a5-1000-4abb-8964-d721b3a1bc37-metrics-certs" pod "132139a5-1000-4abb-8964-d721b3a1bc37" (UID: "132139a5-1000-4abb-8964-d721b3a1bc37"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:34:29.141918 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:29.140012    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:29.206978 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:29.204858    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/132139a5-1000-4abb-8964-d721b3a1bc37-maas-api-tls\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:34:29.206978 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:29.204983    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgq9d\" (UniqueName: \"kubernetes.io/projected/132139a5-1000-4abb-8964-d721b3a1bc37-kube-api-access-tgq9d\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:34:29.206978 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:29.205001    5898 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/132139a5-1000-4abb-8964-d721b3a1bc37-metrics-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:34:29.283537 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:29.283355    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:29.485828 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:29.476787    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:29.637685 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:29.633281    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:29.837677 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:29.837648    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:29.932129 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:29.932095    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-a914e9-777dc7878c-j4drn" event={"ID":"132139a5-1000-4abb-8964-d721b3a1bc37","Type":"ContainerDied","Data":"f7975768c3910fc94ced81bf7d4794f8aee0f7c15571f1be181a7f56fd9d8cb3"}
Oct 30 05:34:29.932308 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:29.932156    5898 scope.go:122] "RemoveContainer" containerID="cdc8942d9db2c701ccb2f4c339c37324bbc7949f0e4e8af925bf7b7b92af8ae9"
Oct 30 05:34:29.932308 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:29.932113    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-a914e9-777dc7878c-j4drn"
Oct 30 05:34:29.990164 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:29.990121    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-a914e9-777dc7878c-j4drn"]
Oct 30 05:34:29.997333 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:29.997300    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-a914e9-777dc7878c-j4drn"]
Oct 30 05:34:30.000301 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:30.000275    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:30.026881 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:30.026846    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:34:31.053087 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:31.053046    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="132139a5-1000-4abb-8964-d721b3a1bc37" path="/var/lib/kubelet/pods/132139a5-1000-4abb-8964-d721b3a1bc37/volumes"
Oct 30 05:34:32.418387 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:32.418338    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:34.480037 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:34.479990    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:36.327634 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:36.327589    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:37.082322 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:37.082274    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:37.260681 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:37.260651    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:37.558388 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:37.558299    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:37.567444 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:37.567413    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js"]
Oct 30 05:34:37.567671 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:37.567648    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js" podUID="fbe41d7a-fb7d-4712-ae76-19d311778e9f" containerName="istio-proxy" containerID="cri-o://7c844efcd3083d3eb575bc2f327eb86dabe53d1af849ad9a084e433834c43176" gracePeriod=30
Oct 30 05:34:37.953819 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:37.953785    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:38.731812 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:38.731782    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:39.126849 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:39.126812    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:39.720412 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:39.720366    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 05:34:39.720597 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:39.720442    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js" podUID="fbe41d7a-fb7d-4712-ae76-19d311778e9f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 05:34:41.312477 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:41.312442    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:42.851197 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:42.851136    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js"
Oct 30 05:34:42.900126 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:42.900096    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbe41d7a-fb7d-4712-ae76-19d311778e9f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbe41d7a-fb7d-4712-ae76-19d311778e9f-workload-certs\") pod \"fbe41d7a-fb7d-4712-ae76-19d311778e9f\" (UID: \"fbe41d7a-fb7d-4712-ae76-19d311778e9f\") "
Oct 30 05:34:42.900293 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:42.900147    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbe41d7a-fb7d-4712-ae76-19d311778e9f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbe41d7a-fb7d-4712-ae76-19d311778e9f-istio-data\") pod \"fbe41d7a-fb7d-4712-ae76-19d311778e9f\" (UID: \"fbe41d7a-fb7d-4712-ae76-19d311778e9f\") "
Oct 30 05:34:42.900293 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:42.900181    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbe41d7a-fb7d-4712-ae76-19d311778e9f-istio-token\" (UniqueName: \"kubernetes.io/projected/fbe41d7a-fb7d-4712-ae76-19d311778e9f-istio-token\") pod \"fbe41d7a-fb7d-4712-ae76-19d311778e9f\" (UID: \"fbe41d7a-fb7d-4712-ae76-19d311778e9f\") "
Oct 30 05:34:42.900420 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:42.900293    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbe41d7a-fb7d-4712-ae76-19d311778e9f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbe41d7a-fb7d-4712-ae76-19d311778e9f-istiod-ca-cert\") pod \"fbe41d7a-fb7d-4712-ae76-19d311778e9f\" (UID: \"fbe41d7a-fb7d-4712-ae76-19d311778e9f\") "
Oct 30 05:34:42.900420 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:42.900349    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbe41d7a-fb7d-4712-ae76-19d311778e9f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbe41d7a-fb7d-4712-ae76-19d311778e9f-workload-socket\") pod \"fbe41d7a-fb7d-4712-ae76-19d311778e9f\" (UID: \"fbe41d7a-fb7d-4712-ae76-19d311778e9f\") "
Oct 30 05:34:42.900420 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:42.900354    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbe41d7a-fb7d-4712-ae76-19d311778e9f-workload-certs" pod "fbe41d7a-fb7d-4712-ae76-19d311778e9f" (UID: "fbe41d7a-fb7d-4712-ae76-19d311778e9f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:34:42.900420 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:42.900402    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fbe41d7a-fb7d-4712-ae76-19d311778e9f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbe41d7a-fb7d-4712-ae76-19d311778e9f-istio-podinfo\") pod \"fbe41d7a-fb7d-4712-ae76-19d311778e9f\" (UID: \"fbe41d7a-fb7d-4712-ae76-19d311778e9f\") "
Oct 30 05:34:42.900635 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:42.900448    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbe41d7a-fb7d-4712-ae76-19d311778e9f-istio-data" pod "fbe41d7a-fb7d-4712-ae76-19d311778e9f" (UID: "fbe41d7a-fb7d-4712-ae76-19d311778e9f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:34:42.900635 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:42.900476    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbe41d7a-fb7d-4712-ae76-19d311778e9f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbe41d7a-fb7d-4712-ae76-19d311778e9f-istio-envoy\") pod \"fbe41d7a-fb7d-4712-ae76-19d311778e9f\" (UID: \"fbe41d7a-fb7d-4712-ae76-19d311778e9f\") "
Oct 30 05:34:42.900635 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:42.900528    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbe41d7a-fb7d-4712-ae76-19d311778e9f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbe41d7a-fb7d-4712-ae76-19d311778e9f-credential-socket\") pod \"fbe41d7a-fb7d-4712-ae76-19d311778e9f\" (UID: \"fbe41d7a-fb7d-4712-ae76-19d311778e9f\") "
Oct 30 05:34:42.900806 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:42.900644    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbe41d7a-fb7d-4712-ae76-19d311778e9f-workload-socket" pod "fbe41d7a-fb7d-4712-ae76-19d311778e9f" (UID: "fbe41d7a-fb7d-4712-ae76-19d311778e9f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:34:42.900806 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:42.900648    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbe41d7a-fb7d-4712-ae76-19d311778e9f-kube-api-access-zzbg2\" (UniqueName: \"kubernetes.io/projected/fbe41d7a-fb7d-4712-ae76-19d311778e9f-kube-api-access-zzbg2\") pod \"fbe41d7a-fb7d-4712-ae76-19d311778e9f\" (UID: \"fbe41d7a-fb7d-4712-ae76-19d311778e9f\") "
Oct 30 05:34:42.901615 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:42.901578    5898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbe41d7a-fb7d-4712-ae76-19d311778e9f-workload-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:34:42.902087 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:42.901622    5898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbe41d7a-fb7d-4712-ae76-19d311778e9f-istio-data\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:34:42.902087 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:42.901641    5898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbe41d7a-fb7d-4712-ae76-19d311778e9f-workload-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:34:42.902087 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:42.901676    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbe41d7a-fb7d-4712-ae76-19d311778e9f-istiod-ca-cert" pod "fbe41d7a-fb7d-4712-ae76-19d311778e9f" (UID: "fbe41d7a-fb7d-4712-ae76-19d311778e9f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:34:42.902634 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:42.902606    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbe41d7a-fb7d-4712-ae76-19d311778e9f-credential-socket" pod "fbe41d7a-fb7d-4712-ae76-19d311778e9f" (UID: "fbe41d7a-fb7d-4712-ae76-19d311778e9f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:34:42.903524 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:42.903501    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbe41d7a-fb7d-4712-ae76-19d311778e9f-istio-token" pod "fbe41d7a-fb7d-4712-ae76-19d311778e9f" (UID: "fbe41d7a-fb7d-4712-ae76-19d311778e9f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:34:42.903670 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:42.903652    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbe41d7a-fb7d-4712-ae76-19d311778e9f-kube-api-access-zzbg2" pod "fbe41d7a-fb7d-4712-ae76-19d311778e9f" (UID: "fbe41d7a-fb7d-4712-ae76-19d311778e9f"). InnerVolumeSpecName "kube-api-access-zzbg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:34:42.903935 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:42.903877    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fbe41d7a-fb7d-4712-ae76-19d311778e9f-istio-podinfo" pod "fbe41d7a-fb7d-4712-ae76-19d311778e9f" (UID: "fbe41d7a-fb7d-4712-ae76-19d311778e9f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:34:42.904046 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:42.904031    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbe41d7a-fb7d-4712-ae76-19d311778e9f-istio-envoy" pod "fbe41d7a-fb7d-4712-ae76-19d311778e9f" (UID: "fbe41d7a-fb7d-4712-ae76-19d311778e9f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:34:42.907164 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:42.907141    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g"]
Oct 30 05:34:42.960266 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:42.946494    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58"]
Oct 30 05:34:43.002790 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:43.002760    5898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbe41d7a-fb7d-4712-ae76-19d311778e9f-istio-token\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:34:43.002790 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:43.002790    5898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbe41d7a-fb7d-4712-ae76-19d311778e9f-istiod-ca-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:34:43.003103 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:43.002799    5898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbe41d7a-fb7d-4712-ae76-19d311778e9f-istio-podinfo\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:34:43.003103 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:43.002807    5898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbe41d7a-fb7d-4712-ae76-19d311778e9f-istio-envoy\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:34:43.003103 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:43.002816    5898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbe41d7a-fb7d-4712-ae76-19d311778e9f-credential-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:34:43.003103 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:43.002827    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzbg2\" (UniqueName: \"kubernetes.io/projected/fbe41d7a-fb7d-4712-ae76-19d311778e9f-kube-api-access-zzbg2\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:34:43.082634 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:43.082603    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:43.233555 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:43.233467    5898 generic.go:356] "Generic (PLEG): container finished" podID="fbe41d7a-fb7d-4712-ae76-19d311778e9f" containerID="7c844efcd3083d3eb575bc2f327eb86dabe53d1af849ad9a084e433834c43176" exitCode=0
Oct 30 05:34:43.233555 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:43.233519    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js"
Oct 30 05:34:43.233814 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:43.233547    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js" event={"ID":"fbe41d7a-fb7d-4712-ae76-19d311778e9f","Type":"ContainerDied","Data":"7c844efcd3083d3eb575bc2f327eb86dabe53d1af849ad9a084e433834c43176"}
Oct 30 05:34:43.233814 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:43.233623    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js" event={"ID":"fbe41d7a-fb7d-4712-ae76-19d311778e9f","Type":"ContainerDied","Data":"b473411c71bdfbe0c2fd237ac6a59f05a14892e4a480f40734a4b8b2c370462a"}
Oct 30 05:34:43.233814 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:43.233653    5898 scope.go:122] "RemoveContainer" containerID="7c844efcd3083d3eb575bc2f327eb86dabe53d1af849ad9a084e433834c43176"
Oct 30 05:34:43.245925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:43.245887    5898 scope.go:122] "RemoveContainer" containerID="7c844efcd3083d3eb575bc2f327eb86dabe53d1af849ad9a084e433834c43176"
Oct 30 05:34:43.246218 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:34:43.246197    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c844efcd3083d3eb575bc2f327eb86dabe53d1af849ad9a084e433834c43176\": container with ID starting with 7c844efcd3083d3eb575bc2f327eb86dabe53d1af849ad9a084e433834c43176 not found: ID does not exist" containerID="7c844efcd3083d3eb575bc2f327eb86dabe53d1af849ad9a084e433834c43176"
Oct 30 05:34:43.246285 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:43.246227    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c844efcd3083d3eb575bc2f327eb86dabe53d1af849ad9a084e433834c43176"} err="failed to get container status \"7c844efcd3083d3eb575bc2f327eb86dabe53d1af849ad9a084e433834c43176\": rpc error: code = NotFound desc = could not find container \"7c844efcd3083d3eb575bc2f327eb86dabe53d1af849ad9a084e433834c43176\": container with ID starting with 7c844efcd3083d3eb575bc2f327eb86dabe53d1af849ad9a084e433834c43176 not found: ID does not exist"
Oct 30 05:34:43.292336 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:43.292299    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js"]
Oct 30 05:34:43.300360 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:43.300325    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-a914e9-openshift-default-7997967f6c-l69js"]
Oct 30 05:34:43.458187 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:43.458152    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:43.564133 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:43.564041    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:44.892413 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:44.892374    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbe41d7a-fb7d-4712-ae76-19d311778e9f" path="/var/lib/kubelet/pods/fbe41d7a-fb7d-4712-ae76-19d311778e9f/volumes"
Oct 30 05:34:44.906619 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:44.906591    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:45.150759 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:45.144430    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:46.115389 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:46.115354    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:47.971624 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:47.971586    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:48.786242 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:48.786207    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:48.940920 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:48.937716    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:49.874454 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:49.874422    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:49.997922 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:49.994273    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:50.138288 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:50.138186    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:50.362004 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:50.361968    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg"]
Oct 30 05:34:50.371485 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:50.363112    5898 cpu_manager.go:414] "containerMap: removing container" podUID="1d865d3e-a6e9-407e-844e-cc75fc7f1354" containerName="main"
Oct 30 05:34:50.371485 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:50.368792    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d865d3e-a6e9-407e-844e-cc75fc7f1354" containerName="main"
Oct 30 05:34:50.371485 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:50.368830    5898 cpu_manager.go:414] "containerMap: removing container" podUID="132139a5-1000-4abb-8964-d721b3a1bc37" containerName="maas-api"
Oct 30 05:34:50.371485 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:50.368840    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="132139a5-1000-4abb-8964-d721b3a1bc37" containerName="maas-api"
Oct 30 05:34:50.371485 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:50.368855    5898 cpu_manager.go:414] "containerMap: removing container" podUID="fbe41d7a-fb7d-4712-ae76-19d311778e9f" containerName="istio-proxy"
Oct 30 05:34:50.371485 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:50.368862    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbe41d7a-fb7d-4712-ae76-19d311778e9f" containerName="istio-proxy"
Oct 30 05:34:50.371485 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:50.368973    5898 cpu_manager.go:414] "containerMap: removing container" podUID="4e8672db-7de8-4aee-8b3b-46ccdc97a211" containerName="payload-pre-processing"
Oct 30 05:34:50.371485 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:50.368983    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e8672db-7de8-4aee-8b3b-46ccdc97a211" containerName="payload-pre-processing"
Oct 30 05:34:50.371485 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:50.369175    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e8672db-7de8-4aee-8b3b-46ccdc97a211" containerName="payload-pre-processing"
Oct 30 05:34:50.371485 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:50.369198    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d865d3e-a6e9-407e-844e-cc75fc7f1354" containerName="main"
Oct 30 05:34:50.371485 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:50.369208    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="132139a5-1000-4abb-8964-d721b3a1bc37" containerName="maas-api"
Oct 30 05:34:50.371485 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:50.369218    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbe41d7a-fb7d-4712-ae76-19d311778e9f" containerName="istio-proxy"
Oct 30 05:34:50.391224 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:50.377179    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg"
Oct 30 05:34:50.433480 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:50.411450    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-674082-openshift-default-dockercfg-g6ls2\""
Oct 30 05:34:50.439414 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:50.439367    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg"]
Oct 30 05:34:50.631947 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:50.621322    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5620415-5516-441d-ab13-3a574d363b59-istio-envoy\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg\" (UID: \"a5620415-5516-441d-ab13-3a574d363b59\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg"
Oct 30 05:34:50.631947 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:50.621450    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfq66\" (UniqueName: \"kubernetes.io/projected/a5620415-5516-441d-ab13-3a574d363b59-kube-api-access-hfq66\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg\" (UID: \"a5620415-5516-441d-ab13-3a574d363b59\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg"
Oct 30 05:34:50.631947 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:50.621487    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5620415-5516-441d-ab13-3a574d363b59-workload-certs\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg\" (UID: \"a5620415-5516-441d-ab13-3a574d363b59\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg"
Oct 30 05:34:50.631947 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:50.621574    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5620415-5516-441d-ab13-3a574d363b59-istio-data\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg\" (UID: \"a5620415-5516-441d-ab13-3a574d363b59\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg"
Oct 30 05:34:50.631947 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:50.621668    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5620415-5516-441d-ab13-3a574d363b59-istio-token\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg\" (UID: \"a5620415-5516-441d-ab13-3a574d363b59\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg"
Oct 30 05:34:50.631947 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:50.621708    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5620415-5516-441d-ab13-3a574d363b59-credential-socket\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg\" (UID: \"a5620415-5516-441d-ab13-3a574d363b59\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg"
Oct 30 05:34:50.631947 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:50.621756    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5620415-5516-441d-ab13-3a574d363b59-istiod-ca-cert\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg\" (UID: \"a5620415-5516-441d-ab13-3a574d363b59\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg"
Oct 30 05:34:50.631947 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:50.621923    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5620415-5516-441d-ab13-3a574d363b59-workload-socket\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg\" (UID: \"a5620415-5516-441d-ab13-3a574d363b59\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg"
Oct 30 05:34:50.631947 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:50.621996    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5620415-5516-441d-ab13-3a574d363b59-istio-podinfo\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg\" (UID: \"a5620415-5516-441d-ab13-3a574d363b59\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg"
Oct 30 05:34:50.723277 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:50.723183    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5620415-5516-441d-ab13-3a574d363b59-istio-token\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg\" (UID: \"a5620415-5516-441d-ab13-3a574d363b59\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg"
Oct 30 05:34:50.723277 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:50.723236    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5620415-5516-441d-ab13-3a574d363b59-credential-socket\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg\" (UID: \"a5620415-5516-441d-ab13-3a574d363b59\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg"
Oct 30 05:34:50.723537 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:50.723424    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5620415-5516-441d-ab13-3a574d363b59-istiod-ca-cert\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg\" (UID: \"a5620415-5516-441d-ab13-3a574d363b59\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg"
Oct 30 05:34:50.723617 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:50.723539    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5620415-5516-441d-ab13-3a574d363b59-workload-socket\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg\" (UID: \"a5620415-5516-441d-ab13-3a574d363b59\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg"
Oct 30 05:34:50.723617 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:50.723603    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5620415-5516-441d-ab13-3a574d363b59-istio-podinfo\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg\" (UID: \"a5620415-5516-441d-ab13-3a574d363b59\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg"
Oct 30 05:34:50.723742 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:50.723656    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5620415-5516-441d-ab13-3a574d363b59-credential-socket\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg\" (UID: \"a5620415-5516-441d-ab13-3a574d363b59\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg"
Oct 30 05:34:50.723742 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:50.723694    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5620415-5516-441d-ab13-3a574d363b59-istio-envoy\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg\" (UID: \"a5620415-5516-441d-ab13-3a574d363b59\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg"
Oct 30 05:34:50.723854 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:50.723773    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfq66\" (UniqueName: \"kubernetes.io/projected/a5620415-5516-441d-ab13-3a574d363b59-kube-api-access-hfq66\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg\" (UID: \"a5620415-5516-441d-ab13-3a574d363b59\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg"
Oct 30 05:34:50.723854 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:50.723806    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5620415-5516-441d-ab13-3a574d363b59-workload-certs\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg\" (UID: \"a5620415-5516-441d-ab13-3a574d363b59\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg"
Oct 30 05:34:50.723987 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:50.723879    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5620415-5516-441d-ab13-3a574d363b59-istio-data\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg\" (UID: \"a5620415-5516-441d-ab13-3a574d363b59\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg"
Oct 30 05:34:50.723987 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:50.723949    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5620415-5516-441d-ab13-3a574d363b59-workload-socket\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg\" (UID: \"a5620415-5516-441d-ab13-3a574d363b59\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg"
Oct 30 05:34:50.724243 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:50.724221    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5620415-5516-441d-ab13-3a574d363b59-istiod-ca-cert\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg\" (UID: \"a5620415-5516-441d-ab13-3a574d363b59\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg"
Oct 30 05:34:50.724361 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:50.724342    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5620415-5516-441d-ab13-3a574d363b59-workload-certs\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg\" (UID: \"a5620415-5516-441d-ab13-3a574d363b59\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg"
Oct 30 05:34:50.724496 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:50.724476    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5620415-5516-441d-ab13-3a574d363b59-istio-data\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg\" (UID: \"a5620415-5516-441d-ab13-3a574d363b59\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg"
Oct 30 05:34:50.726235 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:50.726206    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5620415-5516-441d-ab13-3a574d363b59-istio-podinfo\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg\" (UID: \"a5620415-5516-441d-ab13-3a574d363b59\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg"
Oct 30 05:34:50.726348 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:50.726282    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5620415-5516-441d-ab13-3a574d363b59-istio-envoy\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg\" (UID: \"a5620415-5516-441d-ab13-3a574d363b59\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg"
Oct 30 05:34:50.743949 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:50.743925    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5620415-5516-441d-ab13-3a574d363b59-istio-token\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg\" (UID: \"a5620415-5516-441d-ab13-3a574d363b59\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg"
Oct 30 05:34:50.744043 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:50.743989    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfq66\" (UniqueName: \"kubernetes.io/projected/a5620415-5516-441d-ab13-3a574d363b59-kube-api-access-hfq66\") pod \"e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg\" (UID: \"a5620415-5516-441d-ab13-3a574d363b59\") " pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg"
Oct 30 05:34:50.763353 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:50.763333    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg"
Oct 30 05:34:51.106643 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:51.106615    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg"]
Oct 30 05:34:51.109422 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:34:51.109389    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5620415_5516_441d_ab13_3a574d363b59.slice/crio-40d7dc9c41158cb668586000b7cd8c4dc91c6adc7cc086856af975b244d12e8b WatchSource:0}: Error finding container 40d7dc9c41158cb668586000b7cd8c4dc91c6adc7cc086856af975b244d12e8b: Status 404 returned error can't find the container with id 40d7dc9c41158cb668586000b7cd8c4dc91c6adc7cc086856af975b244d12e8b
Oct 30 05:34:51.112781 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:51.112738    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:34:51.112876 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:51.112826    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:34:51.112942 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:51.112880    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:34:51.354232 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:51.354196    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg" event={"ID":"a5620415-5516-441d-ab13-3a574d363b59","Type":"ContainerStarted","Data":"1fe205a80e215e24adc2495bfd885611e4219226c9476eef1a4178f43f52db7b"}
Oct 30 05:34:51.354407 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:51.354273    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg" event={"ID":"a5620415-5516-441d-ab13-3a574d363b59","Type":"ContainerStarted","Data":"40d7dc9c41158cb668586000b7cd8c4dc91c6adc7cc086856af975b244d12e8b"}
Oct 30 05:34:51.403977 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:51.403854    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg" podStartSLOduration=1.4038345890000001 podStartE2EDuration="1.403834589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:34:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:34:51.397730512 +0000 UTC m=+3020.621550414" watchObservedRunningTime="2026-10-30 05:34:51.403834589 +0000 UTC m=+3020.627654491"
Oct 30 05:34:51.763647 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:51.763571    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg"
Oct 30 05:34:52.016432 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:52.016347    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:52.765308 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:52.765265    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Oct 30 05:34:52.765805 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:52.765319    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg" podUID="a5620415-5516-441d-ab13-3a574d363b59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Oct 30 05:34:53.763659 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:53.763615    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Oct 30 05:34:53.763848 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:53.763668    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg" podUID="a5620415-5516-441d-ab13-3a574d363b59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Oct 30 05:34:54.060627 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:54.060541    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:54.763656 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:54.763624    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Oct 30 05:34:54.763862 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:54.763677    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg" podUID="a5620415-5516-441d-ab13-3a574d363b59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Oct 30 05:34:55.404924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:55.403673    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:55.576464 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:55.576427    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:55.766029 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:55.764802    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Oct 30 05:34:55.766029 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:55.764913    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg" podUID="a5620415-5516-441d-ab13-3a574d363b59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Oct 30 05:34:56.458855 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:56.458814    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:56.766997 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:56.766913    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Oct 30 05:34:56.767257 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:56.767230    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg" podUID="a5620415-5516-441d-ab13-3a574d363b59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Oct 30 05:34:56.869943 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:56.867269    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-889ca9-c6b65f8fc-7jqcd"]
Oct 30 05:34:56.869943 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:56.867540    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-889ca9-c6b65f8fc-7jqcd" podUID="5d3b67cf-5656-4d62-8adc-91df8cdcf394" containerName="maas-api" containerID="cri-o://3153ab5f05a819303fe834d19e1e494243595628b4cf0191e9c78aec14b3628e" gracePeriod=30
Oct 30 05:34:56.978343 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:56.972499    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:57.077546 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:34:57.077310    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:34:57.149844 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:57.149807    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:57.341996 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:57.330460    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:57.488926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:57.472203    5898 generic.go:356] "Generic (PLEG): container finished" podID="5d3b67cf-5656-4d62-8adc-91df8cdcf394" containerID="3153ab5f05a819303fe834d19e1e494243595628b4cf0191e9c78aec14b3628e" exitCode=0
Oct 30 05:34:57.488926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:57.472376    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-889ca9-c6b65f8fc-7jqcd" event={"ID":"5d3b67cf-5656-4d62-8adc-91df8cdcf394","Type":"ContainerDied","Data":"3153ab5f05a819303fe834d19e1e494243595628b4cf0191e9c78aec14b3628e"}
Oct 30 05:34:57.668433 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:57.663303    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49" podUID="05eef4eb-f379-4dc7-a947-123bd85a4968" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 05:34:57.754932 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:57.741067    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58" podUID="c27f3b21-1447-4af9-8d99-c04fa9b23cda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:34:57.781926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:57.769467    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6" podUID="d30a7a86-f1da-451c-81dd-ebdbaeb4f8af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:34:57.781926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:57.775115    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Oct 30 05:34:57.781926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:57.775172    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg" podUID="a5620415-5516-441d-ab13-3a574d363b59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Oct 30 05:34:57.820201 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:57.820154    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6" podUID="47ff272a-ac5c-4f59-bf7b-5c0692d09386" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:34:57.841066 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:57.838867    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:57.954449 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:57.948573    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g" podUID="70294a85-1050-4632-b3bf-c3e44a4c0b41" containerName="main" containerID="cri-o://2bae3ddc4b39eb73f211ad13b176ec58efec42ca6434a0166638f41b8b460e5e" gracePeriod=45
Oct 30 05:34:57.955072 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:57.954678    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:57.992424 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:57.991077    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58" podUID="c27f3b21-1447-4af9-8d99-c04fa9b23cda" containerName="main" containerID="cri-o://ae0c691c6bc69ffdb23781f20e0eedf669f942c27bd990c10a039f10472ce2c7" gracePeriod=45
Oct 30 05:34:58.125268 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.125148    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:58.185169 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.178073    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-889ca9-c6b65f8fc-7jqcd"
Oct 30 05:34:58.266401 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.266371    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d3b67cf-5656-4d62-8adc-91df8cdcf394-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d3b67cf-5656-4d62-8adc-91df8cdcf394-metrics-certs\") pod \"5d3b67cf-5656-4d62-8adc-91df8cdcf394\" (UID: \"5d3b67cf-5656-4d62-8adc-91df8cdcf394\") "
Oct 30 05:34:58.266592 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.266574    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d3b67cf-5656-4d62-8adc-91df8cdcf394-kube-api-access-llbvb\" (UniqueName: \"kubernetes.io/projected/5d3b67cf-5656-4d62-8adc-91df8cdcf394-kube-api-access-llbvb\") pod \"5d3b67cf-5656-4d62-8adc-91df8cdcf394\" (UID: \"5d3b67cf-5656-4d62-8adc-91df8cdcf394\") "
Oct 30 05:34:58.266988 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.266962    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d3b67cf-5656-4d62-8adc-91df8cdcf394-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d3b67cf-5656-4d62-8adc-91df8cdcf394-maas-api-tls\") pod \"5d3b67cf-5656-4d62-8adc-91df8cdcf394\" (UID: \"5d3b67cf-5656-4d62-8adc-91df8cdcf394\") "
Oct 30 05:34:58.271414 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.271386    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d3b67cf-5656-4d62-8adc-91df8cdcf394-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d3b67cf-5656-4d62-8adc-91df8cdcf394-openshift-service-ca\") pod \"5d3b67cf-5656-4d62-8adc-91df8cdcf394\" (UID: \"5d3b67cf-5656-4d62-8adc-91df8cdcf394\") "
Oct 30 05:34:58.278618 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.278587    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d3b67cf-5656-4d62-8adc-91df8cdcf394-maas-api-tls" pod "5d3b67cf-5656-4d62-8adc-91df8cdcf394" (UID: "5d3b67cf-5656-4d62-8adc-91df8cdcf394"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:34:58.278745 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.278617    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d3b67cf-5656-4d62-8adc-91df8cdcf394-metrics-certs" pod "5d3b67cf-5656-4d62-8adc-91df8cdcf394" (UID: "5d3b67cf-5656-4d62-8adc-91df8cdcf394"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:34:58.278745 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.278639    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d3b67cf-5656-4d62-8adc-91df8cdcf394-openshift-service-ca" pod "5d3b67cf-5656-4d62-8adc-91df8cdcf394" (UID: "5d3b67cf-5656-4d62-8adc-91df8cdcf394"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:34:58.278877 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.278827    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d3b67cf-5656-4d62-8adc-91df8cdcf394-kube-api-access-llbvb" pod "5d3b67cf-5656-4d62-8adc-91df8cdcf394" (UID: "5d3b67cf-5656-4d62-8adc-91df8cdcf394"). InnerVolumeSpecName "kube-api-access-llbvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:34:58.289435 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.288277    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:58.337943 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.337737    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g"
Oct 30 05:34:58.406320 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.381098    5898 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d3b67cf-5656-4d62-8adc-91df8cdcf394-openshift-service-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:34:58.427424 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.425027    5898 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d3b67cf-5656-4d62-8adc-91df8cdcf394-metrics-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:34:58.427424 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.425070    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llbvb\" (UniqueName: \"kubernetes.io/projected/5d3b67cf-5656-4d62-8adc-91df8cdcf394-kube-api-access-llbvb\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:34:58.427424 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.425086    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d3b67cf-5656-4d62-8adc-91df8cdcf394-maas-api-tls\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:34:58.458973 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.458808    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58"
Oct 30 05:34:58.482601 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.482513    5898 generic.go:356] "Generic (PLEG): container finished" podID="c27f3b21-1447-4af9-8d99-c04fa9b23cda" containerID="ae0c691c6bc69ffdb23781f20e0eedf669f942c27bd990c10a039f10472ce2c7" exitCode=0
Oct 30 05:34:58.482601 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.482561    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58"
Oct 30 05:34:58.482846 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.482652    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58" event={"ID":"c27f3b21-1447-4af9-8d99-c04fa9b23cda","Type":"ContainerDied","Data":"ae0c691c6bc69ffdb23781f20e0eedf669f942c27bd990c10a039f10472ce2c7"}
Oct 30 05:34:58.482846 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.482685    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58" event={"ID":"c27f3b21-1447-4af9-8d99-c04fa9b23cda","Type":"ContainerDied","Data":"1a6cfdcd44c131c15223e5be73a5a7107d08f26d2e869ec8b6df374f7c9cfd32"}
Oct 30 05:34:58.482846 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.482708    5898 scope.go:122] "RemoveContainer" containerID="ae0c691c6bc69ffdb23781f20e0eedf669f942c27bd990c10a039f10472ce2c7"
Oct 30 05:34:58.484794 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.484771    5898 generic.go:356] "Generic (PLEG): container finished" podID="70294a85-1050-4632-b3bf-c3e44a4c0b41" containerID="2bae3ddc4b39eb73f211ad13b176ec58efec42ca6434a0166638f41b8b460e5e" exitCode=0
Oct 30 05:34:58.484983 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.484815    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g"
Oct 30 05:34:58.484983 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.484856    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g" event={"ID":"70294a85-1050-4632-b3bf-c3e44a4c0b41","Type":"ContainerDied","Data":"2bae3ddc4b39eb73f211ad13b176ec58efec42ca6434a0166638f41b8b460e5e"}
Oct 30 05:34:58.484983 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.484884    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g" event={"ID":"70294a85-1050-4632-b3bf-c3e44a4c0b41","Type":"ContainerDied","Data":"1cad023cb4608d77f4afa45b0ef20859c9171bb4d2139559ad1f8fa283b49245"}
Oct 30 05:34:58.486686 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.486670    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-889ca9-c6b65f8fc-7jqcd"
Oct 30 05:34:58.487691 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.486704    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-889ca9-c6b65f8fc-7jqcd" event={"ID":"5d3b67cf-5656-4d62-8adc-91df8cdcf394","Type":"ContainerDied","Data":"cabce4aeefa8bc80fdf33b52e59c6f7a8f7c4112874b6bdc698119e6c09f272c"}
Oct 30 05:34:58.497628 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.497600    5898 scope.go:122] "RemoveContainer" containerID="ae0c691c6bc69ffdb23781f20e0eedf669f942c27bd990c10a039f10472ce2c7"
Oct 30 05:34:58.500363 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:34:58.498356    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae0c691c6bc69ffdb23781f20e0eedf669f942c27bd990c10a039f10472ce2c7\": container with ID starting with ae0c691c6bc69ffdb23781f20e0eedf669f942c27bd990c10a039f10472ce2c7 not found: ID does not exist" containerID="ae0c691c6bc69ffdb23781f20e0eedf669f942c27bd990c10a039f10472ce2c7"
Oct 30 05:34:58.500363 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.498430    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae0c691c6bc69ffdb23781f20e0eedf669f942c27bd990c10a039f10472ce2c7"} err="failed to get container status \"ae0c691c6bc69ffdb23781f20e0eedf669f942c27bd990c10a039f10472ce2c7\": rpc error: code = NotFound desc = could not find container \"ae0c691c6bc69ffdb23781f20e0eedf669f942c27bd990c10a039f10472ce2c7\": container with ID starting with ae0c691c6bc69ffdb23781f20e0eedf669f942c27bd990c10a039f10472ce2c7 not found: ID does not exist"
Oct 30 05:34:58.500363 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.498454    5898 scope.go:122] "RemoveContainer" containerID="2bae3ddc4b39eb73f211ad13b176ec58efec42ca6434a0166638f41b8b460e5e"
Oct 30 05:34:58.531506 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.531426    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70294a85-1050-4632-b3bf-c3e44a4c0b41-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70294a85-1050-4632-b3bf-c3e44a4c0b41-tmp-dir\") pod \"70294a85-1050-4632-b3bf-c3e44a4c0b41\" (UID: \"70294a85-1050-4632-b3bf-c3e44a4c0b41\") "
Oct 30 05:34:58.531506 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.531496    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c27f3b21-1447-4af9-8d99-c04fa9b23cda-kube-api-access-92kgx\" (UniqueName: \"kubernetes.io/projected/c27f3b21-1447-4af9-8d99-c04fa9b23cda-kube-api-access-92kgx\") pod \"c27f3b21-1447-4af9-8d99-c04fa9b23cda\" (UID: \"c27f3b21-1447-4af9-8d99-c04fa9b23cda\") "
Oct 30 05:34:58.531693 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.531535    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70294a85-1050-4632-b3bf-c3e44a4c0b41-dshm\" (UniqueName: \"kubernetes.io/empty-dir/70294a85-1050-4632-b3bf-c3e44a4c0b41-dshm\") pod \"70294a85-1050-4632-b3bf-c3e44a4c0b41\" (UID: \"70294a85-1050-4632-b3bf-c3e44a4c0b41\") "
Oct 30 05:34:58.531693 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.531579    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70294a85-1050-4632-b3bf-c3e44a4c0b41-home\" (UniqueName: \"kubernetes.io/empty-dir/70294a85-1050-4632-b3bf-c3e44a4c0b41-home\") pod \"70294a85-1050-4632-b3bf-c3e44a4c0b41\" (UID: \"70294a85-1050-4632-b3bf-c3e44a4c0b41\") "
Oct 30 05:34:58.531693 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.531604    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70294a85-1050-4632-b3bf-c3e44a4c0b41-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70294a85-1050-4632-b3bf-c3e44a4c0b41-model-cache\") pod \"70294a85-1050-4632-b3bf-c3e44a4c0b41\" (UID: \"70294a85-1050-4632-b3bf-c3e44a4c0b41\") "
Oct 30 05:34:58.531892 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.531787    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c27f3b21-1447-4af9-8d99-c04fa9b23cda-tls-certs\" (UniqueName: \"kubernetes.io/secret/c27f3b21-1447-4af9-8d99-c04fa9b23cda-tls-certs\") pod \"c27f3b21-1447-4af9-8d99-c04fa9b23cda\" (UID: \"c27f3b21-1447-4af9-8d99-c04fa9b23cda\") "
Oct 30 05:34:58.531892 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.531821    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70294a85-1050-4632-b3bf-c3e44a4c0b41-tls-certs\" (UniqueName: \"kubernetes.io/secret/70294a85-1050-4632-b3bf-c3e44a4c0b41-tls-certs\") pod \"70294a85-1050-4632-b3bf-c3e44a4c0b41\" (UID: \"70294a85-1050-4632-b3bf-c3e44a4c0b41\") "
Oct 30 05:34:58.531892 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.531863    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c27f3b21-1447-4af9-8d99-c04fa9b23cda-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c27f3b21-1447-4af9-8d99-c04fa9b23cda-model-cache\") pod \"c27f3b21-1447-4af9-8d99-c04fa9b23cda\" (UID: \"c27f3b21-1447-4af9-8d99-c04fa9b23cda\") "
Oct 30 05:34:58.532085 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.531959    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70294a85-1050-4632-b3bf-c3e44a4c0b41-kube-api-access-xk2cm\" (UniqueName: \"kubernetes.io/projected/70294a85-1050-4632-b3bf-c3e44a4c0b41-kube-api-access-xk2cm\") pod \"70294a85-1050-4632-b3bf-c3e44a4c0b41\" (UID: \"70294a85-1050-4632-b3bf-c3e44a4c0b41\") "
Oct 30 05:34:58.532085 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.532069    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c27f3b21-1447-4af9-8d99-c04fa9b23cda-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c27f3b21-1447-4af9-8d99-c04fa9b23cda-tmp-dir\") pod \"c27f3b21-1447-4af9-8d99-c04fa9b23cda\" (UID: \"c27f3b21-1447-4af9-8d99-c04fa9b23cda\") "
Oct 30 05:34:58.532203 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.532107    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c27f3b21-1447-4af9-8d99-c04fa9b23cda-home\" (UniqueName: \"kubernetes.io/empty-dir/c27f3b21-1447-4af9-8d99-c04fa9b23cda-home\") pod \"c27f3b21-1447-4af9-8d99-c04fa9b23cda\" (UID: \"c27f3b21-1447-4af9-8d99-c04fa9b23cda\") "
Oct 30 05:34:58.532203 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.532136    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c27f3b21-1447-4af9-8d99-c04fa9b23cda-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c27f3b21-1447-4af9-8d99-c04fa9b23cda-dshm\") pod \"c27f3b21-1447-4af9-8d99-c04fa9b23cda\" (UID: \"c27f3b21-1447-4af9-8d99-c04fa9b23cda\") "
Oct 30 05:34:58.532616 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.532593    5898 scope.go:122] "RemoveContainer" containerID="2bae3ddc4b39eb73f211ad13b176ec58efec42ca6434a0166638f41b8b460e5e"
Oct 30 05:34:58.533410 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.533371    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70294a85-1050-4632-b3bf-c3e44a4c0b41-tmp-dir" pod "70294a85-1050-4632-b3bf-c3e44a4c0b41" (UID: "70294a85-1050-4632-b3bf-c3e44a4c0b41"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:34:58.533538 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:34:58.533472    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bae3ddc4b39eb73f211ad13b176ec58efec42ca6434a0166638f41b8b460e5e\": container with ID starting with 2bae3ddc4b39eb73f211ad13b176ec58efec42ca6434a0166638f41b8b460e5e not found: ID does not exist" containerID="2bae3ddc4b39eb73f211ad13b176ec58efec42ca6434a0166638f41b8b460e5e"
Oct 30 05:34:58.533538 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.533505    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bae3ddc4b39eb73f211ad13b176ec58efec42ca6434a0166638f41b8b460e5e"} err="failed to get container status \"2bae3ddc4b39eb73f211ad13b176ec58efec42ca6434a0166638f41b8b460e5e\": rpc error: code = NotFound desc = could not find container \"2bae3ddc4b39eb73f211ad13b176ec58efec42ca6434a0166638f41b8b460e5e\": container with ID starting with 2bae3ddc4b39eb73f211ad13b176ec58efec42ca6434a0166638f41b8b460e5e not found: ID does not exist"
Oct 30 05:34:58.533538 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.533528    5898 scope.go:122] "RemoveContainer" containerID="3153ab5f05a819303fe834d19e1e494243595628b4cf0191e9c78aec14b3628e"
Oct 30 05:34:58.534075 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.534048    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70294a85-1050-4632-b3bf-c3e44a4c0b41-home" pod "70294a85-1050-4632-b3bf-c3e44a4c0b41" (UID: "70294a85-1050-4632-b3bf-c3e44a4c0b41"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:34:58.534217 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.534196    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70294a85-1050-4632-b3bf-c3e44a4c0b41-model-cache" pod "70294a85-1050-4632-b3bf-c3e44a4c0b41" (UID: "70294a85-1050-4632-b3bf-c3e44a4c0b41"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:34:58.535277 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.534990    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c27f3b21-1447-4af9-8d99-c04fa9b23cda-model-cache" pod "c27f3b21-1447-4af9-8d99-c04fa9b23cda" (UID: "c27f3b21-1447-4af9-8d99-c04fa9b23cda"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:34:58.535464 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.535365    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c27f3b21-1447-4af9-8d99-c04fa9b23cda-home" pod "c27f3b21-1447-4af9-8d99-c04fa9b23cda" (UID: "c27f3b21-1447-4af9-8d99-c04fa9b23cda"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:34:58.535464 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.535440    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c27f3b21-1447-4af9-8d99-c04fa9b23cda-tmp-dir" pod "c27f3b21-1447-4af9-8d99-c04fa9b23cda" (UID: "c27f3b21-1447-4af9-8d99-c04fa9b23cda"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:34:58.537290 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.537257    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70294a85-1050-4632-b3bf-c3e44a4c0b41-dshm" pod "70294a85-1050-4632-b3bf-c3e44a4c0b41" (UID: "70294a85-1050-4632-b3bf-c3e44a4c0b41"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:34:58.537946 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.537877    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c27f3b21-1447-4af9-8d99-c04fa9b23cda-kube-api-access-92kgx" pod "c27f3b21-1447-4af9-8d99-c04fa9b23cda" (UID: "c27f3b21-1447-4af9-8d99-c04fa9b23cda"). InnerVolumeSpecName "kube-api-access-92kgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:34:58.538479 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.538452    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c27f3b21-1447-4af9-8d99-c04fa9b23cda-dshm" pod "c27f3b21-1447-4af9-8d99-c04fa9b23cda" (UID: "c27f3b21-1447-4af9-8d99-c04fa9b23cda"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:34:58.538886 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.538860    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70294a85-1050-4632-b3bf-c3e44a4c0b41-kube-api-access-xk2cm" pod "70294a85-1050-4632-b3bf-c3e44a4c0b41" (UID: "70294a85-1050-4632-b3bf-c3e44a4c0b41"). InnerVolumeSpecName "kube-api-access-xk2cm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:34:58.539141 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.539118    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70294a85-1050-4632-b3bf-c3e44a4c0b41-tls-certs" pod "70294a85-1050-4632-b3bf-c3e44a4c0b41" (UID: "70294a85-1050-4632-b3bf-c3e44a4c0b41"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:34:58.539699 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.539673    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c27f3b21-1447-4af9-8d99-c04fa9b23cda-tls-certs" pod "c27f3b21-1447-4af9-8d99-c04fa9b23cda" (UID: "c27f3b21-1447-4af9-8d99-c04fa9b23cda"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:34:58.596578 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.596545    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-889ca9-c6b65f8fc-7jqcd"]
Oct 30 05:34:58.605939 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.605892    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-889ca9-c6b65f8fc-7jqcd"]
Oct 30 05:34:58.634764 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.634733    5898 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c27f3b21-1447-4af9-8d99-c04fa9b23cda-tls-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:34:58.634764 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.634761    5898 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70294a85-1050-4632-b3bf-c3e44a4c0b41-tls-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:34:58.634764 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.634770    5898 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c27f3b21-1447-4af9-8d99-c04fa9b23cda-model-cache\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:34:58.635034 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.634779    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk2cm\" (UniqueName: \"kubernetes.io/projected/70294a85-1050-4632-b3bf-c3e44a4c0b41-kube-api-access-xk2cm\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:34:58.635034 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.634788    5898 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c27f3b21-1447-4af9-8d99-c04fa9b23cda-tmp-dir\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:34:58.635034 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.634799    5898 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c27f3b21-1447-4af9-8d99-c04fa9b23cda-home\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:34:58.635034 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.634806    5898 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c27f3b21-1447-4af9-8d99-c04fa9b23cda-dshm\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:34:58.635034 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.634813    5898 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70294a85-1050-4632-b3bf-c3e44a4c0b41-tmp-dir\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:34:58.635034 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.634821    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92kgx\" (UniqueName: \"kubernetes.io/projected/c27f3b21-1447-4af9-8d99-c04fa9b23cda-kube-api-access-92kgx\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:34:58.635034 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.634828    5898 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70294a85-1050-4632-b3bf-c3e44a4c0b41-dshm\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:34:58.635034 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.634837    5898 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70294a85-1050-4632-b3bf-c3e44a4c0b41-home\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:34:58.635034 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.634844    5898 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70294a85-1050-4632-b3bf-c3e44a4c0b41-model-cache\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:34:58.824844 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.824799    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:34:58.850215 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.850183    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g"]
Oct 30 05:34:58.863271 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.863242    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-83e83e/distinct-2-w6-83e83e-kserve-58ff78d94d-cwz7g"]
Oct 30 05:34:58.898621 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.898588    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d3b67cf-5656-4d62-8adc-91df8cdcf394" path="/var/lib/kubelet/pods/5d3b67cf-5656-4d62-8adc-91df8cdcf394/volumes"
Oct 30 05:34:58.899302 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.899275    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70294a85-1050-4632-b3bf-c3e44a4c0b41" path="/var/lib/kubelet/pods/70294a85-1050-4632-b3bf-c3e44a4c0b41/volumes"
Oct 30 05:34:58.900554 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.900528    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58"]
Oct 30 05:34:58.907719 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.907691    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-83e83e/simulator-w6-83e83e-kserve-f76fb6869-qjk58"]
Oct 30 05:34:58.968945 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.968861    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg"
Oct 30 05:34:58.969418 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.969395    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg"
Oct 30 05:34:58.970189 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:34:58.970164    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg"
Oct 30 05:35:00.423444 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:00.423412    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:00.893094 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:00.893061    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c27f3b21-1447-4af9-8d99-c04fa9b23cda" path="/var/lib/kubelet/pods/c27f3b21-1447-4af9-8d99-c04fa9b23cda/volumes"
Oct 30 05:35:02.992613 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:02.992576    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:05.222616 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:05.222561    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:05.430644 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:05.430613    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:05.563864 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:05.561624    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:05.704329 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:05.704294    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:05.863110 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:05.863072    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4"]
Oct 30 05:35:05.863346 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:05.863320    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4" podUID="3bc29954-1331-4bdf-9d6c-57e5894a5947" containerName="istio-proxy" containerID="cri-o://40d770fe1a0221210344d1f60c830b9d991dbea16f042644d676602cfd14bf67" gracePeriod=30
Oct 30 05:35:05.870152 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:05.870109    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:06.032929 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:06.032339    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:06.087924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:06.085954    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd"]
Oct 30 05:35:06.087924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:06.086228    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd" podUID="ced35944-e749-49e0-97ac-806b824c601c" containerName="istio-proxy" containerID="cri-o://232c13f86c349523d3d1cc3640458822ef4d34ac3ae3a1ed54dba41656301e03" gracePeriod=30
Oct 30 05:35:06.275137 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:06.233241    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:08.310540 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:08.310494    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:10.262431 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:10.262399    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:11.144386 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.144331    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4"
Oct 30 05:35:11.320110 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.320082    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3bc29954-1331-4bdf-9d6c-57e5894a5947-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bc29954-1331-4bdf-9d6c-57e5894a5947-istio-podinfo\") pod \"3bc29954-1331-4bdf-9d6c-57e5894a5947\" (UID: \"3bc29954-1331-4bdf-9d6c-57e5894a5947\") "
Oct 30 05:35:11.320520 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.320257    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bc29954-1331-4bdf-9d6c-57e5894a5947-kube-api-access-n422v\" (UniqueName: \"kubernetes.io/projected/3bc29954-1331-4bdf-9d6c-57e5894a5947-kube-api-access-n422v\") pod \"3bc29954-1331-4bdf-9d6c-57e5894a5947\" (UID: \"3bc29954-1331-4bdf-9d6c-57e5894a5947\") "
Oct 30 05:35:11.320520 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.320292    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bc29954-1331-4bdf-9d6c-57e5894a5947-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc29954-1331-4bdf-9d6c-57e5894a5947-workload-socket\") pod \"3bc29954-1331-4bdf-9d6c-57e5894a5947\" (UID: \"3bc29954-1331-4bdf-9d6c-57e5894a5947\") "
Oct 30 05:35:11.320520 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.320354    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bc29954-1331-4bdf-9d6c-57e5894a5947-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc29954-1331-4bdf-9d6c-57e5894a5947-credential-socket\") pod \"3bc29954-1331-4bdf-9d6c-57e5894a5947\" (UID: \"3bc29954-1331-4bdf-9d6c-57e5894a5947\") "
Oct 30 05:35:11.320520 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.320396    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bc29954-1331-4bdf-9d6c-57e5894a5947-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bc29954-1331-4bdf-9d6c-57e5894a5947-workload-certs\") pod \"3bc29954-1331-4bdf-9d6c-57e5894a5947\" (UID: \"3bc29954-1331-4bdf-9d6c-57e5894a5947\") "
Oct 30 05:35:11.320520 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.320508    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bc29954-1331-4bdf-9d6c-57e5894a5947-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bc29954-1331-4bdf-9d6c-57e5894a5947-istio-data\") pod \"3bc29954-1331-4bdf-9d6c-57e5894a5947\" (UID: \"3bc29954-1331-4bdf-9d6c-57e5894a5947\") "
Oct 30 05:35:11.320826 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.320568    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bc29954-1331-4bdf-9d6c-57e5894a5947-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bc29954-1331-4bdf-9d6c-57e5894a5947-istio-envoy\") pod \"3bc29954-1331-4bdf-9d6c-57e5894a5947\" (UID: \"3bc29954-1331-4bdf-9d6c-57e5894a5947\") "
Oct 30 05:35:11.320826 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.320597    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bc29954-1331-4bdf-9d6c-57e5894a5947-istio-token\" (UniqueName: \"kubernetes.io/projected/3bc29954-1331-4bdf-9d6c-57e5894a5947-istio-token\") pod \"3bc29954-1331-4bdf-9d6c-57e5894a5947\" (UID: \"3bc29954-1331-4bdf-9d6c-57e5894a5947\") "
Oct 30 05:35:11.320826 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.320641    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bc29954-1331-4bdf-9d6c-57e5894a5947-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bc29954-1331-4bdf-9d6c-57e5894a5947-istiod-ca-cert\") pod \"3bc29954-1331-4bdf-9d6c-57e5894a5947\" (UID: \"3bc29954-1331-4bdf-9d6c-57e5894a5947\") "
Oct 30 05:35:11.320826 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.320786    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bc29954-1331-4bdf-9d6c-57e5894a5947-workload-certs" pod "3bc29954-1331-4bdf-9d6c-57e5894a5947" (UID: "3bc29954-1331-4bdf-9d6c-57e5894a5947"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:35:11.320826 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.320798    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bc29954-1331-4bdf-9d6c-57e5894a5947-workload-socket" pod "3bc29954-1331-4bdf-9d6c-57e5894a5947" (UID: "3bc29954-1331-4bdf-9d6c-57e5894a5947"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:35:11.320826 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.320811    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bc29954-1331-4bdf-9d6c-57e5894a5947-istio-data" pod "3bc29954-1331-4bdf-9d6c-57e5894a5947" (UID: "3bc29954-1331-4bdf-9d6c-57e5894a5947"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:35:11.321179 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.320924    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bc29954-1331-4bdf-9d6c-57e5894a5947-credential-socket" pod "3bc29954-1331-4bdf-9d6c-57e5894a5947" (UID: "3bc29954-1331-4bdf-9d6c-57e5894a5947"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:35:11.321403 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.321372    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bc29954-1331-4bdf-9d6c-57e5894a5947-istiod-ca-cert" pod "3bc29954-1331-4bdf-9d6c-57e5894a5947" (UID: "3bc29954-1331-4bdf-9d6c-57e5894a5947"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:35:11.321537 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.321479    5898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc29954-1331-4bdf-9d6c-57e5894a5947-workload-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:35:11.321537 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.321502    5898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc29954-1331-4bdf-9d6c-57e5894a5947-credential-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:35:11.321537 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.321518    5898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bc29954-1331-4bdf-9d6c-57e5894a5947-workload-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:35:11.321537 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.321535    5898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bc29954-1331-4bdf-9d6c-57e5894a5947-istio-data\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:35:11.322686 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.322665    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3bc29954-1331-4bdf-9d6c-57e5894a5947-istio-podinfo" pod "3bc29954-1331-4bdf-9d6c-57e5894a5947" (UID: "3bc29954-1331-4bdf-9d6c-57e5894a5947"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:35:11.322956 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.322894    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bc29954-1331-4bdf-9d6c-57e5894a5947-kube-api-access-n422v" pod "3bc29954-1331-4bdf-9d6c-57e5894a5947" (UID: "3bc29954-1331-4bdf-9d6c-57e5894a5947"). InnerVolumeSpecName "kube-api-access-n422v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:35:11.323213 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.323192    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bc29954-1331-4bdf-9d6c-57e5894a5947-istio-token" pod "3bc29954-1331-4bdf-9d6c-57e5894a5947" (UID: "3bc29954-1331-4bdf-9d6c-57e5894a5947"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:35:11.323281 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.323230    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bc29954-1331-4bdf-9d6c-57e5894a5947-istio-envoy" pod "3bc29954-1331-4bdf-9d6c-57e5894a5947" (UID: "3bc29954-1331-4bdf-9d6c-57e5894a5947"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:35:11.423457 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.423415    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n422v\" (UniqueName: \"kubernetes.io/projected/3bc29954-1331-4bdf-9d6c-57e5894a5947-kube-api-access-n422v\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:35:11.423457 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.423458    5898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bc29954-1331-4bdf-9d6c-57e5894a5947-istio-envoy\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:35:11.423652 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.423474    5898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bc29954-1331-4bdf-9d6c-57e5894a5947-istio-token\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:35:11.423652 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.423487    5898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bc29954-1331-4bdf-9d6c-57e5894a5947-istiod-ca-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:35:11.423652 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.423500    5898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bc29954-1331-4bdf-9d6c-57e5894a5947-istio-podinfo\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:35:11.557381 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.557355    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd"
Oct 30 05:35:11.667724 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.667640    5898 generic.go:356] "Generic (PLEG): container finished" podID="ced35944-e749-49e0-97ac-806b824c601c" containerID="232c13f86c349523d3d1cc3640458822ef4d34ac3ae3a1ed54dba41656301e03" exitCode=0
Oct 30 05:35:11.667724 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.667689    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd"
Oct 30 05:35:11.668021 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.667719    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd" event={"ID":"ced35944-e749-49e0-97ac-806b824c601c","Type":"ContainerDied","Data":"232c13f86c349523d3d1cc3640458822ef4d34ac3ae3a1ed54dba41656301e03"}
Oct 30 05:35:11.668021 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.667766    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd" event={"ID":"ced35944-e749-49e0-97ac-806b824c601c","Type":"ContainerDied","Data":"433ff2fe513d5690e04c04e7c293e03bc8710e79900c859bf921d51fdc5f8427"}
Oct 30 05:35:11.668021 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.667792    5898 scope.go:122] "RemoveContainer" containerID="232c13f86c349523d3d1cc3640458822ef4d34ac3ae3a1ed54dba41656301e03"
Oct 30 05:35:11.669124 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.669103    5898 generic.go:356] "Generic (PLEG): container finished" podID="3bc29954-1331-4bdf-9d6c-57e5894a5947" containerID="40d770fe1a0221210344d1f60c830b9d991dbea16f042644d676602cfd14bf67" exitCode=0
Oct 30 05:35:11.669240 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.669146    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4" event={"ID":"3bc29954-1331-4bdf-9d6c-57e5894a5947","Type":"ContainerDied","Data":"40d770fe1a0221210344d1f60c830b9d991dbea16f042644d676602cfd14bf67"}
Oct 30 05:35:11.669240 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.669186    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4" event={"ID":"3bc29954-1331-4bdf-9d6c-57e5894a5947","Type":"ContainerDied","Data":"fd1a63addc6502ad91f4197ff8d90fba711885265ddd20d0f1abf1737d424cab"}
Oct 30 05:35:11.669240 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.669150    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4"
Oct 30 05:35:11.679988 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.679965    5898 scope.go:122] "RemoveContainer" containerID="232c13f86c349523d3d1cc3640458822ef4d34ac3ae3a1ed54dba41656301e03"
Oct 30 05:35:11.680243 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:35:11.680226    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"232c13f86c349523d3d1cc3640458822ef4d34ac3ae3a1ed54dba41656301e03\": container with ID starting with 232c13f86c349523d3d1cc3640458822ef4d34ac3ae3a1ed54dba41656301e03 not found: ID does not exist" containerID="232c13f86c349523d3d1cc3640458822ef4d34ac3ae3a1ed54dba41656301e03"
Oct 30 05:35:11.680289 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.680252    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"232c13f86c349523d3d1cc3640458822ef4d34ac3ae3a1ed54dba41656301e03"} err="failed to get container status \"232c13f86c349523d3d1cc3640458822ef4d34ac3ae3a1ed54dba41656301e03\": rpc error: code = NotFound desc = could not find container \"232c13f86c349523d3d1cc3640458822ef4d34ac3ae3a1ed54dba41656301e03\": container with ID starting with 232c13f86c349523d3d1cc3640458822ef4d34ac3ae3a1ed54dba41656301e03 not found: ID does not exist"
Oct 30 05:35:11.680289 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.680270    5898 scope.go:122] "RemoveContainer" containerID="40d770fe1a0221210344d1f60c830b9d991dbea16f042644d676602cfd14bf67"
Oct 30 05:35:11.689605 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.689590    5898 scope.go:122] "RemoveContainer" containerID="40d770fe1a0221210344d1f60c830b9d991dbea16f042644d676602cfd14bf67"
Oct 30 05:35:11.689858 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:35:11.689840    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40d770fe1a0221210344d1f60c830b9d991dbea16f042644d676602cfd14bf67\": container with ID starting with 40d770fe1a0221210344d1f60c830b9d991dbea16f042644d676602cfd14bf67 not found: ID does not exist" containerID="40d770fe1a0221210344d1f60c830b9d991dbea16f042644d676602cfd14bf67"
Oct 30 05:35:11.689944 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.689865    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40d770fe1a0221210344d1f60c830b9d991dbea16f042644d676602cfd14bf67"} err="failed to get container status \"40d770fe1a0221210344d1f60c830b9d991dbea16f042644d676602cfd14bf67\": rpc error: code = NotFound desc = could not find container \"40d770fe1a0221210344d1f60c830b9d991dbea16f042644d676602cfd14bf67\": container with ID starting with 40d770fe1a0221210344d1f60c830b9d991dbea16f042644d676602cfd14bf67 not found: ID does not exist"
Oct 30 05:35:11.723642 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.723616    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4"]
Oct 30 05:35:11.726866 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.726836    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ced35944-e749-49e0-97ac-806b824c601c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ced35944-e749-49e0-97ac-806b824c601c-istio-data\") pod \"ced35944-e749-49e0-97ac-806b824c601c\" (UID: \"ced35944-e749-49e0-97ac-806b824c601c\") "
Oct 30 05:35:11.727023 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.726980    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ced35944-e749-49e0-97ac-806b824c601c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ced35944-e749-49e0-97ac-806b824c601c-workload-socket\") pod \"ced35944-e749-49e0-97ac-806b824c601c\" (UID: \"ced35944-e749-49e0-97ac-806b824c601c\") "
Oct 30 05:35:11.727208 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.727086    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ced35944-e749-49e0-97ac-806b824c601c-kube-api-access-5scr4\" (UniqueName: \"kubernetes.io/projected/ced35944-e749-49e0-97ac-806b824c601c-kube-api-access-5scr4\") pod \"ced35944-e749-49e0-97ac-806b824c601c\" (UID: \"ced35944-e749-49e0-97ac-806b824c601c\") "
Oct 30 05:35:11.727208 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.727144    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ced35944-e749-49e0-97ac-806b824c601c-istio-data" pod "ced35944-e749-49e0-97ac-806b824c601c" (UID: "ced35944-e749-49e0-97ac-806b824c601c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:35:11.727208 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.727135    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ced35944-e749-49e0-97ac-806b824c601c-istio-token\" (UniqueName: \"kubernetes.io/projected/ced35944-e749-49e0-97ac-806b824c601c-istio-token\") pod \"ced35944-e749-49e0-97ac-806b824c601c\" (UID: \"ced35944-e749-49e0-97ac-806b824c601c\") "
Oct 30 05:35:11.727558 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.727281    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ced35944-e749-49e0-97ac-806b824c601c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ced35944-e749-49e0-97ac-806b824c601c-workload-certs\") pod \"ced35944-e749-49e0-97ac-806b824c601c\" (UID: \"ced35944-e749-49e0-97ac-806b824c601c\") "
Oct 30 05:35:11.727558 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.727352    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ced35944-e749-49e0-97ac-806b824c601c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ced35944-e749-49e0-97ac-806b824c601c-istiod-ca-cert\") pod \"ced35944-e749-49e0-97ac-806b824c601c\" (UID: \"ced35944-e749-49e0-97ac-806b824c601c\") "
Oct 30 05:35:11.727558 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.727438    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ced35944-e749-49e0-97ac-806b824c601c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ced35944-e749-49e0-97ac-806b824c601c-istio-envoy\") pod \"ced35944-e749-49e0-97ac-806b824c601c\" (UID: \"ced35944-e749-49e0-97ac-806b824c601c\") "
Oct 30 05:35:11.728214 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.727571    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ced35944-e749-49e0-97ac-806b824c601c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ced35944-e749-49e0-97ac-806b824c601c-istio-podinfo\") pod \"ced35944-e749-49e0-97ac-806b824c601c\" (UID: \"ced35944-e749-49e0-97ac-806b824c601c\") "
Oct 30 05:35:11.728214 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.727657    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ced35944-e749-49e0-97ac-806b824c601c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ced35944-e749-49e0-97ac-806b824c601c-credential-socket\") pod \"ced35944-e749-49e0-97ac-806b824c601c\" (UID: \"ced35944-e749-49e0-97ac-806b824c601c\") "
Oct 30 05:35:11.728418 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.728382    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ced35944-e749-49e0-97ac-806b824c601c-workload-certs" pod "ced35944-e749-49e0-97ac-806b824c601c" (UID: "ced35944-e749-49e0-97ac-806b824c601c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:35:11.728540 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.728494    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ced35944-e749-49e0-97ac-806b824c601c-istiod-ca-cert" pod "ced35944-e749-49e0-97ac-806b824c601c" (UID: "ced35944-e749-49e0-97ac-806b824c601c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:35:11.728663 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.728638    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ced35944-e749-49e0-97ac-806b824c601c-credential-socket" pod "ced35944-e749-49e0-97ac-806b824c601c" (UID: "ced35944-e749-49e0-97ac-806b824c601c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:35:11.728782 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.728759    5898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ced35944-e749-49e0-97ac-806b824c601c-credential-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:35:11.728952 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.728784    5898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ced35944-e749-49e0-97ac-806b824c601c-istio-data\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:35:11.728952 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.728805    5898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ced35944-e749-49e0-97ac-806b824c601c-workload-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:35:11.728952 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.728819    5898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ced35944-e749-49e0-97ac-806b824c601c-istiod-ca-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:35:11.728952 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.728835    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ced35944-e749-49e0-97ac-806b824c601c-workload-socket" pod "ced35944-e749-49e0-97ac-806b824c601c" (UID: "ced35944-e749-49e0-97ac-806b824c601c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:35:11.730432 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.730402    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ced35944-e749-49e0-97ac-806b824c601c-istio-token" pod "ced35944-e749-49e0-97ac-806b824c601c" (UID: "ced35944-e749-49e0-97ac-806b824c601c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:35:11.730979 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.730950    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ced35944-e749-49e0-97ac-806b824c601c-istio-envoy" pod "ced35944-e749-49e0-97ac-806b824c601c" (UID: "ced35944-e749-49e0-97ac-806b824c601c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:35:11.731089 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.730994    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ced35944-e749-49e0-97ac-806b824c601c-istio-podinfo" pod "ced35944-e749-49e0-97ac-806b824c601c" (UID: "ced35944-e749-49e0-97ac-806b824c601c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:35:11.731266 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.731246    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ced35944-e749-49e0-97ac-806b824c601c-kube-api-access-5scr4" pod "ced35944-e749-49e0-97ac-806b824c601c" (UID: "ced35944-e749-49e0-97ac-806b824c601c"). InnerVolumeSpecName "kube-api-access-5scr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:35:11.733132 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.733110    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-262c10-openshift-default-6f56f59fcf-9khb4"]
Oct 30 05:35:11.836935 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.835711    5898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ced35944-e749-49e0-97ac-806b824c601c-istio-podinfo\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:35:11.836935 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.835755    5898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ced35944-e749-49e0-97ac-806b824c601c-workload-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:35:11.836935 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.835771    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5scr4\" (UniqueName: \"kubernetes.io/projected/ced35944-e749-49e0-97ac-806b824c601c-kube-api-access-5scr4\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:35:11.836935 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.835789    5898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ced35944-e749-49e0-97ac-806b824c601c-istio-token\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:35:11.836935 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:11.835803    5898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ced35944-e749-49e0-97ac-806b824c601c-istio-envoy\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:35:12.090751 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:12.090721    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd"]
Oct 30 05:35:12.104263 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:12.104232    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-889ca9-openshift-default-6dbd4b87f5-blvkd"]
Oct 30 05:35:12.177980 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:12.177949    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:12.891367 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:12.891333    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bc29954-1331-4bdf-9d6c-57e5894a5947" path="/var/lib/kubelet/pods/3bc29954-1331-4bdf-9d6c-57e5894a5947/volumes"
Oct 30 05:35:12.892033 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:12.892006    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ced35944-e749-49e0-97ac-806b824c601c" path="/var/lib/kubelet/pods/ced35944-e749-49e0-97ac-806b824c601c/volumes"
Oct 30 05:35:13.743638 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:13.743602    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:14.000659 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:14.000578    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:14.128229 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:14.128194    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:14.266990 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:14.266885    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:14.393041 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:14.393005    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:14.565988 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:14.563568    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:14.852892 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:14.852865    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:17.029371 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:17.029332    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:17.143537 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:17.143493    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:18.830277 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:18.830233    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:19.106569 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:19.106491    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:19.842735 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:19.842685    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:19.982854 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:19.982761    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:20.266863 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:20.266780    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:20.795998 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:20.795927    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:20.925609 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:20.925569    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:22.747968 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:22.744170    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:22.925873 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:22.925833    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:24.751993 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:24.751960    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:24.892265 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:24.892232    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:25.018980 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:25.017854    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:26.136107 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:26.136075    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss"]
Oct 30 05:35:26.136541 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:26.136276    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss" podUID="5e630691-ed49-45ba-ac72-67ef295e80f3" containerName="istio-proxy" containerID="cri-o://cd08ba50e7496e9421154554582d82cb6cbcbe93039b6b211b09d4405eb5be86" gracePeriod=30
Oct 30 05:35:26.140524 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:26.140494    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs"]
Oct 30 05:35:26.140742 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:26.140718    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs" podUID="aa609b95-6fbc-4cb6-9153-97e49232e302" containerName="istio-proxy" containerID="cri-o://5d1cf8101476826b36d8e1390abad24b9543c23aafadf4445db029e31647ee55" gracePeriod=30
Oct 30 05:35:26.819479 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:35:26.819441    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:35:27.425411 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:27.425363    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:29.278461 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:29.278425    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:29.388496 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:29.388446    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:29.633990 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:29.633948    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:31.465556 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.465530    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss"
Oct 30 05:35:31.471993 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.471969    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e630691-ed49-45ba-ac72-67ef295e80f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e630691-ed49-45ba-ac72-67ef295e80f3-istio-envoy\") pod \"5e630691-ed49-45ba-ac72-67ef295e80f3\" (UID: \"5e630691-ed49-45ba-ac72-67ef295e80f3\") "
Oct 30 05:35:31.472115 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.472031    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e630691-ed49-45ba-ac72-67ef295e80f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e630691-ed49-45ba-ac72-67ef295e80f3-workload-socket\") pod \"5e630691-ed49-45ba-ac72-67ef295e80f3\" (UID: \"5e630691-ed49-45ba-ac72-67ef295e80f3\") "
Oct 30 05:35:31.472206 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.472194    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e630691-ed49-45ba-ac72-67ef295e80f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e630691-ed49-45ba-ac72-67ef295e80f3-istio-data\") pod \"5e630691-ed49-45ba-ac72-67ef295e80f3\" (UID: \"5e630691-ed49-45ba-ac72-67ef295e80f3\") "
Oct 30 05:35:31.472254 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.472241    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5e630691-ed49-45ba-ac72-67ef295e80f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e630691-ed49-45ba-ac72-67ef295e80f3-istio-podinfo\") pod \"5e630691-ed49-45ba-ac72-67ef295e80f3\" (UID: \"5e630691-ed49-45ba-ac72-67ef295e80f3\") "
Oct 30 05:35:31.472312 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.472273    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e630691-ed49-45ba-ac72-67ef295e80f3-istio-token\" (UniqueName: \"kubernetes.io/projected/5e630691-ed49-45ba-ac72-67ef295e80f3-istio-token\") pod \"5e630691-ed49-45ba-ac72-67ef295e80f3\" (UID: \"5e630691-ed49-45ba-ac72-67ef295e80f3\") "
Oct 30 05:35:31.472359 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.472302    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e630691-ed49-45ba-ac72-67ef295e80f3-workload-socket" pod "5e630691-ed49-45ba-ac72-67ef295e80f3" (UID: "5e630691-ed49-45ba-ac72-67ef295e80f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:35:31.472359 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.472309    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs"
Oct 30 05:35:31.472472 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.472401    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e630691-ed49-45ba-ac72-67ef295e80f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e630691-ed49-45ba-ac72-67ef295e80f3-workload-certs\") pod \"5e630691-ed49-45ba-ac72-67ef295e80f3\" (UID: \"5e630691-ed49-45ba-ac72-67ef295e80f3\") "
Oct 30 05:35:31.472530 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.472467    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e630691-ed49-45ba-ac72-67ef295e80f3-istio-data" pod "5e630691-ed49-45ba-ac72-67ef295e80f3" (UID: "5e630691-ed49-45ba-ac72-67ef295e80f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:35:31.472530 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.472492    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e630691-ed49-45ba-ac72-67ef295e80f3-kube-api-access-829pt\" (UniqueName: \"kubernetes.io/projected/5e630691-ed49-45ba-ac72-67ef295e80f3-kube-api-access-829pt\") pod \"5e630691-ed49-45ba-ac72-67ef295e80f3\" (UID: \"5e630691-ed49-45ba-ac72-67ef295e80f3\") "
Oct 30 05:35:31.472648 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.472591    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e630691-ed49-45ba-ac72-67ef295e80f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e630691-ed49-45ba-ac72-67ef295e80f3-credential-socket\") pod \"5e630691-ed49-45ba-ac72-67ef295e80f3\" (UID: \"5e630691-ed49-45ba-ac72-67ef295e80f3\") "
Oct 30 05:35:31.472648 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.472645    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e630691-ed49-45ba-ac72-67ef295e80f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e630691-ed49-45ba-ac72-67ef295e80f3-istiod-ca-cert\") pod \"5e630691-ed49-45ba-ac72-67ef295e80f3\" (UID: \"5e630691-ed49-45ba-ac72-67ef295e80f3\") "
Oct 30 05:35:31.472762 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.472651    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e630691-ed49-45ba-ac72-67ef295e80f3-workload-certs" pod "5e630691-ed49-45ba-ac72-67ef295e80f3" (UID: "5e630691-ed49-45ba-ac72-67ef295e80f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:35:31.472858 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.472835    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e630691-ed49-45ba-ac72-67ef295e80f3-credential-socket" pod "5e630691-ed49-45ba-ac72-67ef295e80f3" (UID: "5e630691-ed49-45ba-ac72-67ef295e80f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:35:31.473036 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.473017    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e630691-ed49-45ba-ac72-67ef295e80f3-istiod-ca-cert" pod "5e630691-ed49-45ba-ac72-67ef295e80f3" (UID: "5e630691-ed49-45ba-ac72-67ef295e80f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:35:31.473320 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.473297    5898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e630691-ed49-45ba-ac72-67ef295e80f3-credential-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:35:31.473477 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.473456    5898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e630691-ed49-45ba-ac72-67ef295e80f3-istiod-ca-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:35:31.473579 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.473482    5898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e630691-ed49-45ba-ac72-67ef295e80f3-workload-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:35:31.473579 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.473496    5898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e630691-ed49-45ba-ac72-67ef295e80f3-istio-data\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:35:31.473579 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.473511    5898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e630691-ed49-45ba-ac72-67ef295e80f3-workload-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:35:31.474430 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.474404    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e630691-ed49-45ba-ac72-67ef295e80f3-istio-envoy" pod "5e630691-ed49-45ba-ac72-67ef295e80f3" (UID: "5e630691-ed49-45ba-ac72-67ef295e80f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:35:31.474529 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.474420    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e630691-ed49-45ba-ac72-67ef295e80f3-istio-podinfo" pod "5e630691-ed49-45ba-ac72-67ef295e80f3" (UID: "5e630691-ed49-45ba-ac72-67ef295e80f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:35:31.474529 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.474493    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e630691-ed49-45ba-ac72-67ef295e80f3-kube-api-access-829pt" pod "5e630691-ed49-45ba-ac72-67ef295e80f3" (UID: "5e630691-ed49-45ba-ac72-67ef295e80f3"). InnerVolumeSpecName "kube-api-access-829pt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:35:31.474689 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.474672    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e630691-ed49-45ba-ac72-67ef295e80f3-istio-token" pod "5e630691-ed49-45ba-ac72-67ef295e80f3" (UID: "5e630691-ed49-45ba-ac72-67ef295e80f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:35:31.574568 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.574516    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa609b95-6fbc-4cb6-9153-97e49232e302-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa609b95-6fbc-4cb6-9153-97e49232e302-credential-socket\") pod \"aa609b95-6fbc-4cb6-9153-97e49232e302\" (UID: \"aa609b95-6fbc-4cb6-9153-97e49232e302\") "
Oct 30 05:35:31.575043 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.575005    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa609b95-6fbc-4cb6-9153-97e49232e302-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa609b95-6fbc-4cb6-9153-97e49232e302-istio-data\") pod \"aa609b95-6fbc-4cb6-9153-97e49232e302\" (UID: \"aa609b95-6fbc-4cb6-9153-97e49232e302\") "
Oct 30 05:35:31.575311 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.575291    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa609b95-6fbc-4cb6-9153-97e49232e302-istio-token\" (UniqueName: \"kubernetes.io/projected/aa609b95-6fbc-4cb6-9153-97e49232e302-istio-token\") pod \"aa609b95-6fbc-4cb6-9153-97e49232e302\" (UID: \"aa609b95-6fbc-4cb6-9153-97e49232e302\") "
Oct 30 05:35:31.575507 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.575489    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa609b95-6fbc-4cb6-9153-97e49232e302-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa609b95-6fbc-4cb6-9153-97e49232e302-istio-podinfo\") pod \"aa609b95-6fbc-4cb6-9153-97e49232e302\" (UID: \"aa609b95-6fbc-4cb6-9153-97e49232e302\") "
Oct 30 05:35:31.575727 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.575708    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa609b95-6fbc-4cb6-9153-97e49232e302-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa609b95-6fbc-4cb6-9153-97e49232e302-workload-certs\") pod \"aa609b95-6fbc-4cb6-9153-97e49232e302\" (UID: \"aa609b95-6fbc-4cb6-9153-97e49232e302\") "
Oct 30 05:35:31.575921 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.575887    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa609b95-6fbc-4cb6-9153-97e49232e302-kube-api-access-9l6hl\" (UniqueName: \"kubernetes.io/projected/aa609b95-6fbc-4cb6-9153-97e49232e302-kube-api-access-9l6hl\") pod \"aa609b95-6fbc-4cb6-9153-97e49232e302\" (UID: \"aa609b95-6fbc-4cb6-9153-97e49232e302\") "
Oct 30 05:35:31.576115 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.576080    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa609b95-6fbc-4cb6-9153-97e49232e302-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa609b95-6fbc-4cb6-9153-97e49232e302-istiod-ca-cert\") pod \"aa609b95-6fbc-4cb6-9153-97e49232e302\" (UID: \"aa609b95-6fbc-4cb6-9153-97e49232e302\") "
Oct 30 05:35:31.576323 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.576305    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa609b95-6fbc-4cb6-9153-97e49232e302-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa609b95-6fbc-4cb6-9153-97e49232e302-workload-socket\") pod \"aa609b95-6fbc-4cb6-9153-97e49232e302\" (UID: \"aa609b95-6fbc-4cb6-9153-97e49232e302\") "
Oct 30 05:35:31.576579 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.576561    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa609b95-6fbc-4cb6-9153-97e49232e302-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa609b95-6fbc-4cb6-9153-97e49232e302-istio-envoy\") pod \"aa609b95-6fbc-4cb6-9153-97e49232e302\" (UID: \"aa609b95-6fbc-4cb6-9153-97e49232e302\") "
Oct 30 05:35:31.577764 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.577738    5898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e630691-ed49-45ba-ac72-67ef295e80f3-istio-podinfo\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:35:31.577966 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.577769    5898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e630691-ed49-45ba-ac72-67ef295e80f3-istio-token\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:35:31.577966 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.577817    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-829pt\" (UniqueName: \"kubernetes.io/projected/5e630691-ed49-45ba-ac72-67ef295e80f3-kube-api-access-829pt\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:35:31.577966 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.577834    5898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e630691-ed49-45ba-ac72-67ef295e80f3-istio-envoy\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:35:31.583097 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.583007    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa609b95-6fbc-4cb6-9153-97e49232e302-istio-envoy" pod "aa609b95-6fbc-4cb6-9153-97e49232e302" (UID: "aa609b95-6fbc-4cb6-9153-97e49232e302"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:35:31.583500 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.583454    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa609b95-6fbc-4cb6-9153-97e49232e302-istio-data" pod "aa609b95-6fbc-4cb6-9153-97e49232e302" (UID: "aa609b95-6fbc-4cb6-9153-97e49232e302"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:35:31.583661 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.583472    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa609b95-6fbc-4cb6-9153-97e49232e302-workload-certs" pod "aa609b95-6fbc-4cb6-9153-97e49232e302" (UID: "aa609b95-6fbc-4cb6-9153-97e49232e302"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:35:31.583988 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.583862    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa609b95-6fbc-4cb6-9153-97e49232e302-istiod-ca-cert" pod "aa609b95-6fbc-4cb6-9153-97e49232e302" (UID: "aa609b95-6fbc-4cb6-9153-97e49232e302"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:35:31.584086 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.584001    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa609b95-6fbc-4cb6-9153-97e49232e302-credential-socket" pod "aa609b95-6fbc-4cb6-9153-97e49232e302" (UID: "aa609b95-6fbc-4cb6-9153-97e49232e302"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:35:31.584229 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.584196    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa609b95-6fbc-4cb6-9153-97e49232e302-workload-socket" pod "aa609b95-6fbc-4cb6-9153-97e49232e302" (UID: "aa609b95-6fbc-4cb6-9153-97e49232e302"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:35:31.584395 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.584360    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa609b95-6fbc-4cb6-9153-97e49232e302-istio-podinfo" pod "aa609b95-6fbc-4cb6-9153-97e49232e302" (UID: "aa609b95-6fbc-4cb6-9153-97e49232e302"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:35:31.586357 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.586322    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa609b95-6fbc-4cb6-9153-97e49232e302-kube-api-access-9l6hl" pod "aa609b95-6fbc-4cb6-9153-97e49232e302" (UID: "aa609b95-6fbc-4cb6-9153-97e49232e302"). InnerVolumeSpecName "kube-api-access-9l6hl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:35:31.586858 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.586838    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa609b95-6fbc-4cb6-9153-97e49232e302-istio-token" pod "aa609b95-6fbc-4cb6-9153-97e49232e302" (UID: "aa609b95-6fbc-4cb6-9153-97e49232e302"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:35:31.628396 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.628366    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:31.679616 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.679550    5898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa609b95-6fbc-4cb6-9153-97e49232e302-workload-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:35:31.679616 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.679578    5898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa609b95-6fbc-4cb6-9153-97e49232e302-istio-envoy\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:35:31.679616 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.679588    5898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa609b95-6fbc-4cb6-9153-97e49232e302-credential-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:35:31.679616 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.679596    5898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa609b95-6fbc-4cb6-9153-97e49232e302-istio-data\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:35:31.679616 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.679606    5898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa609b95-6fbc-4cb6-9153-97e49232e302-istio-token\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:35:31.679616 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.679614    5898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa609b95-6fbc-4cb6-9153-97e49232e302-istio-podinfo\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:35:31.679931 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.679622    5898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa609b95-6fbc-4cb6-9153-97e49232e302-workload-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:35:31.679931 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.679630    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l6hl\" (UniqueName: \"kubernetes.io/projected/aa609b95-6fbc-4cb6-9153-97e49232e302-kube-api-access-9l6hl\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:35:31.679931 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.679638    5898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa609b95-6fbc-4cb6-9153-97e49232e302-istiod-ca-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:35:31.904414 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.904376    5898 generic.go:356] "Generic (PLEG): container finished" podID="5e630691-ed49-45ba-ac72-67ef295e80f3" containerID="cd08ba50e7496e9421154554582d82cb6cbcbe93039b6b211b09d4405eb5be86" exitCode=0
Oct 30 05:35:31.904609 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.904418    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss"
Oct 30 05:35:31.904609 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.904423    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss" event={"ID":"5e630691-ed49-45ba-ac72-67ef295e80f3","Type":"ContainerDied","Data":"cd08ba50e7496e9421154554582d82cb6cbcbe93039b6b211b09d4405eb5be86"}
Oct 30 05:35:31.904609 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.904475    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss" event={"ID":"5e630691-ed49-45ba-ac72-67ef295e80f3","Type":"ContainerDied","Data":"49348ed659d5f54a3773de72a145cf4510cceb477191b99015e0541d6a0ced1c"}
Oct 30 05:35:31.904609 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.904496    5898 scope.go:122] "RemoveContainer" containerID="cd08ba50e7496e9421154554582d82cb6cbcbe93039b6b211b09d4405eb5be86"
Oct 30 05:35:31.906216 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.906191    5898 generic.go:356] "Generic (PLEG): container finished" podID="aa609b95-6fbc-4cb6-9153-97e49232e302" containerID="5d1cf8101476826b36d8e1390abad24b9543c23aafadf4445db029e31647ee55" exitCode=0
Oct 30 05:35:31.906339 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.906253    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs"
Oct 30 05:35:31.906339 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.906273    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs" event={"ID":"aa609b95-6fbc-4cb6-9153-97e49232e302","Type":"ContainerDied","Data":"5d1cf8101476826b36d8e1390abad24b9543c23aafadf4445db029e31647ee55"}
Oct 30 05:35:31.906339 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.906325    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs" event={"ID":"aa609b95-6fbc-4cb6-9153-97e49232e302","Type":"ContainerDied","Data":"763e58b745e4562cd8e7f3a022f4a9f3bc9dd038a0599fb21671fede7e1221c4"}
Oct 30 05:35:31.916712 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.916696    5898 scope.go:122] "RemoveContainer" containerID="cd08ba50e7496e9421154554582d82cb6cbcbe93039b6b211b09d4405eb5be86"
Oct 30 05:35:31.917015 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:35:31.916997    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd08ba50e7496e9421154554582d82cb6cbcbe93039b6b211b09d4405eb5be86\": container with ID starting with cd08ba50e7496e9421154554582d82cb6cbcbe93039b6b211b09d4405eb5be86 not found: ID does not exist" containerID="cd08ba50e7496e9421154554582d82cb6cbcbe93039b6b211b09d4405eb5be86"
Oct 30 05:35:31.917079 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.917022    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd08ba50e7496e9421154554582d82cb6cbcbe93039b6b211b09d4405eb5be86"} err="failed to get container status \"cd08ba50e7496e9421154554582d82cb6cbcbe93039b6b211b09d4405eb5be86\": rpc error: code = NotFound desc = could not find container \"cd08ba50e7496e9421154554582d82cb6cbcbe93039b6b211b09d4405eb5be86\": container with ID starting with cd08ba50e7496e9421154554582d82cb6cbcbe93039b6b211b09d4405eb5be86 not found: ID does not exist"
Oct 30 05:35:31.917079 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.917037    5898 scope.go:122] "RemoveContainer" containerID="5d1cf8101476826b36d8e1390abad24b9543c23aafadf4445db029e31647ee55"
Oct 30 05:35:31.933080 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.933053    5898 scope.go:122] "RemoveContainer" containerID="5d1cf8101476826b36d8e1390abad24b9543c23aafadf4445db029e31647ee55"
Oct 30 05:35:31.933414 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:35:31.933392    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d1cf8101476826b36d8e1390abad24b9543c23aafadf4445db029e31647ee55\": container with ID starting with 5d1cf8101476826b36d8e1390abad24b9543c23aafadf4445db029e31647ee55 not found: ID does not exist" containerID="5d1cf8101476826b36d8e1390abad24b9543c23aafadf4445db029e31647ee55"
Oct 30 05:35:31.933497 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.933420    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d1cf8101476826b36d8e1390abad24b9543c23aafadf4445db029e31647ee55"} err="failed to get container status \"5d1cf8101476826b36d8e1390abad24b9543c23aafadf4445db029e31647ee55\": rpc error: code = NotFound desc = could not find container \"5d1cf8101476826b36d8e1390abad24b9543c23aafadf4445db029e31647ee55\": container with ID starting with 5d1cf8101476826b36d8e1390abad24b9543c23aafadf4445db029e31647ee55 not found: ID does not exist"
Oct 30 05:35:31.956514 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.956483    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss"]
Oct 30 05:35:31.963817 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.963794    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-8lxss"]
Oct 30 05:35:31.996943 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:31.996890    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs"]
Oct 30 05:35:32.000868 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:32.000843    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-jkqvs"]
Oct 30 05:35:32.893103 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:32.893071    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e630691-ed49-45ba-ac72-67ef295e80f3" path="/var/lib/kubelet/pods/5e630691-ed49-45ba-ac72-67ef295e80f3/volumes"
Oct 30 05:35:32.893601 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:32.893586    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa609b95-6fbc-4cb6-9153-97e49232e302" path="/var/lib/kubelet/pods/aa609b95-6fbc-4cb6-9153-97e49232e302/volumes"
Oct 30 05:35:33.707783 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:33.707752    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:36.093329 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:36.093267    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:37.881646 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:37.881610    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:39.150801 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:39.150766    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:35:39.873604 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:39.873569    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:41.878736 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:41.878679    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:43.634381 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:43.634348    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:43.854933 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:43.854882    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:46.115203 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:46.115165    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:46.840914 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:46.840867    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:47.071962 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:47.071929    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:47.230840 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:47.230764    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:47.629304 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:47.629266    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:49.089403 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:49.089372    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:50.640892 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:50.640854    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:51.262339 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:51.262265    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:51.499298 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:51.499245    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:51.606335 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:51.606297    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:51.715009 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:51.714973    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:52.014373 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:52.014217    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:52.334718 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:52.334680    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:52.461039 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:52.455193    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:52.857503 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:52.857472    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:53.525785 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:53.525748    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:53.631955 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:53.631919    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:53.872540 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:53.872501    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq"]
Oct 30 05:35:53.873167 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:53.872749    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq" podUID="fecce31e-a6fa-443f-aedf-327ef9710f2a" containerName="istio-proxy" containerID="cri-o://706fa53890f44692b093fe30d77e461d56d14e307071755ea21e09e28bf2d70b" gracePeriod=30
Oct 30 05:35:53.903048 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:53.903018    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:54.569743 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:54.569715    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:55.897849 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:55.897806    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:56.070801 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:56.070764    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:56.818458 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:35:56.818424    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:35:57.275670 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:57.275638    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:59.152474 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:59.152405    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq"
Oct 30 05:35:59.172061 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:59.172028    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:59.263013 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:59.262433    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fecce31e-a6fa-443f-aedf-327ef9710f2a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fecce31e-a6fa-443f-aedf-327ef9710f2a-istio-envoy\") pod \"fecce31e-a6fa-443f-aedf-327ef9710f2a\" (UID: \"fecce31e-a6fa-443f-aedf-327ef9710f2a\") "
Oct 30 05:35:59.263013 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:59.262558    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fecce31e-a6fa-443f-aedf-327ef9710f2a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fecce31e-a6fa-443f-aedf-327ef9710f2a-istio-podinfo\") pod \"fecce31e-a6fa-443f-aedf-327ef9710f2a\" (UID: \"fecce31e-a6fa-443f-aedf-327ef9710f2a\") "
Oct 30 05:35:59.263013 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:59.262611    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fecce31e-a6fa-443f-aedf-327ef9710f2a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fecce31e-a6fa-443f-aedf-327ef9710f2a-istiod-ca-cert\") pod \"fecce31e-a6fa-443f-aedf-327ef9710f2a\" (UID: \"fecce31e-a6fa-443f-aedf-327ef9710f2a\") "
Oct 30 05:35:59.263013 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:59.262717    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fecce31e-a6fa-443f-aedf-327ef9710f2a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fecce31e-a6fa-443f-aedf-327ef9710f2a-workload-socket\") pod \"fecce31e-a6fa-443f-aedf-327ef9710f2a\" (UID: \"fecce31e-a6fa-443f-aedf-327ef9710f2a\") "
Oct 30 05:35:59.263013 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:59.262791    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fecce31e-a6fa-443f-aedf-327ef9710f2a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fecce31e-a6fa-443f-aedf-327ef9710f2a-credential-socket\") pod \"fecce31e-a6fa-443f-aedf-327ef9710f2a\" (UID: \"fecce31e-a6fa-443f-aedf-327ef9710f2a\") "
Oct 30 05:35:59.263619 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:59.263381    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fecce31e-a6fa-443f-aedf-327ef9710f2a-istiod-ca-cert" pod "fecce31e-a6fa-443f-aedf-327ef9710f2a" (UID: "fecce31e-a6fa-443f-aedf-327ef9710f2a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:35:59.264798 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:59.263692    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fecce31e-a6fa-443f-aedf-327ef9710f2a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fecce31e-a6fa-443f-aedf-327ef9710f2a-workload-certs\") pod \"fecce31e-a6fa-443f-aedf-327ef9710f2a\" (UID: \"fecce31e-a6fa-443f-aedf-327ef9710f2a\") "
Oct 30 05:35:59.264798 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:59.263776    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fecce31e-a6fa-443f-aedf-327ef9710f2a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fecce31e-a6fa-443f-aedf-327ef9710f2a-istio-data\") pod \"fecce31e-a6fa-443f-aedf-327ef9710f2a\" (UID: \"fecce31e-a6fa-443f-aedf-327ef9710f2a\") "
Oct 30 05:35:59.264798 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:59.263828    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fecce31e-a6fa-443f-aedf-327ef9710f2a-kube-api-access-l8djx\" (UniqueName: \"kubernetes.io/projected/fecce31e-a6fa-443f-aedf-327ef9710f2a-kube-api-access-l8djx\") pod \"fecce31e-a6fa-443f-aedf-327ef9710f2a\" (UID: \"fecce31e-a6fa-443f-aedf-327ef9710f2a\") "
Oct 30 05:35:59.264798 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:59.264420    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fecce31e-a6fa-443f-aedf-327ef9710f2a-workload-socket" pod "fecce31e-a6fa-443f-aedf-327ef9710f2a" (UID: "fecce31e-a6fa-443f-aedf-327ef9710f2a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:35:59.264798 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:59.264615    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fecce31e-a6fa-443f-aedf-327ef9710f2a-credential-socket" pod "fecce31e-a6fa-443f-aedf-327ef9710f2a" (UID: "fecce31e-a6fa-443f-aedf-327ef9710f2a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:35:59.264798 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:59.264794    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fecce31e-a6fa-443f-aedf-327ef9710f2a-workload-certs" pod "fecce31e-a6fa-443f-aedf-327ef9710f2a" (UID: "fecce31e-a6fa-443f-aedf-327ef9710f2a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:35:59.265329 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:59.264860    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fecce31e-a6fa-443f-aedf-327ef9710f2a-istio-data" pod "fecce31e-a6fa-443f-aedf-327ef9710f2a" (UID: "fecce31e-a6fa-443f-aedf-327ef9710f2a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:35:59.272360 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:59.271562    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fecce31e-a6fa-443f-aedf-327ef9710f2a-istio-token\" (UniqueName: \"kubernetes.io/projected/fecce31e-a6fa-443f-aedf-327ef9710f2a-istio-token\") pod \"fecce31e-a6fa-443f-aedf-327ef9710f2a\" (UID: \"fecce31e-a6fa-443f-aedf-327ef9710f2a\") "
Oct 30 05:35:59.272360 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:59.272347    5898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fecce31e-a6fa-443f-aedf-327ef9710f2a-workload-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:35:59.272562 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:59.272373    5898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fecce31e-a6fa-443f-aedf-327ef9710f2a-credential-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:35:59.272562 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:59.272389    5898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fecce31e-a6fa-443f-aedf-327ef9710f2a-workload-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:35:59.272562 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:59.272402    5898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fecce31e-a6fa-443f-aedf-327ef9710f2a-istio-data\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:35:59.272562 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:59.272414    5898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fecce31e-a6fa-443f-aedf-327ef9710f2a-istiod-ca-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:35:59.275215 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:59.275163    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fecce31e-a6fa-443f-aedf-327ef9710f2a-istio-podinfo" pod "fecce31e-a6fa-443f-aedf-327ef9710f2a" (UID: "fecce31e-a6fa-443f-aedf-327ef9710f2a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:35:59.286985 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:59.276534    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fecce31e-a6fa-443f-aedf-327ef9710f2a-istio-envoy" pod "fecce31e-a6fa-443f-aedf-327ef9710f2a" (UID: "fecce31e-a6fa-443f-aedf-327ef9710f2a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:35:59.286985 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:59.276655    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fecce31e-a6fa-443f-aedf-327ef9710f2a-kube-api-access-l8djx" pod "fecce31e-a6fa-443f-aedf-327ef9710f2a" (UID: "fecce31e-a6fa-443f-aedf-327ef9710f2a"). InnerVolumeSpecName "kube-api-access-l8djx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:35:59.286985 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:59.278211    5898 generic.go:356] "Generic (PLEG): container finished" podID="fecce31e-a6fa-443f-aedf-327ef9710f2a" containerID="706fa53890f44692b093fe30d77e461d56d14e307071755ea21e09e28bf2d70b" exitCode=0
Oct 30 05:35:59.286985 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:59.278364    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq" event={"ID":"fecce31e-a6fa-443f-aedf-327ef9710f2a","Type":"ContainerDied","Data":"706fa53890f44692b093fe30d77e461d56d14e307071755ea21e09e28bf2d70b"}
Oct 30 05:35:59.286985 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:59.278413    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq" event={"ID":"fecce31e-a6fa-443f-aedf-327ef9710f2a","Type":"ContainerDied","Data":"574c55e467113f34a19fc72828fcf6abdf63b2b3787e1dee02ea019ca907f3a1"}
Oct 30 05:35:59.286985 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:59.278443    5898 scope.go:122] "RemoveContainer" containerID="706fa53890f44692b093fe30d77e461d56d14e307071755ea21e09e28bf2d70b"
Oct 30 05:35:59.286985 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:59.278604    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq"
Oct 30 05:35:59.287704 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:59.287004    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fecce31e-a6fa-443f-aedf-327ef9710f2a-istio-token" pod "fecce31e-a6fa-443f-aedf-327ef9710f2a" (UID: "fecce31e-a6fa-443f-aedf-327ef9710f2a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:35:59.316485 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:59.316462    5898 scope.go:122] "RemoveContainer" containerID="706fa53890f44692b093fe30d77e461d56d14e307071755ea21e09e28bf2d70b"
Oct 30 05:35:59.317024 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:35:59.316996    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"706fa53890f44692b093fe30d77e461d56d14e307071755ea21e09e28bf2d70b\": container with ID starting with 706fa53890f44692b093fe30d77e461d56d14e307071755ea21e09e28bf2d70b not found: ID does not exist" containerID="706fa53890f44692b093fe30d77e461d56d14e307071755ea21e09e28bf2d70b"
Oct 30 05:35:59.317120 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:59.317028    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"706fa53890f44692b093fe30d77e461d56d14e307071755ea21e09e28bf2d70b"} err="failed to get container status \"706fa53890f44692b093fe30d77e461d56d14e307071755ea21e09e28bf2d70b\": rpc error: code = NotFound desc = could not find container \"706fa53890f44692b093fe30d77e461d56d14e307071755ea21e09e28bf2d70b\": container with ID starting with 706fa53890f44692b093fe30d77e461d56d14e307071755ea21e09e28bf2d70b not found: ID does not exist"
Oct 30 05:35:59.373421 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:59.373392    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8djx\" (UniqueName: \"kubernetes.io/projected/fecce31e-a6fa-443f-aedf-327ef9710f2a-kube-api-access-l8djx\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:35:59.373421 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:59.373423    5898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fecce31e-a6fa-443f-aedf-327ef9710f2a-istio-token\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:35:59.373580 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:59.373437    5898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fecce31e-a6fa-443f-aedf-327ef9710f2a-istio-envoy\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:35:59.373580 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:59.373449    5898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fecce31e-a6fa-443f-aedf-327ef9710f2a-istio-podinfo\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:35:59.414109 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:59.411415    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:59.561923 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:59.552350    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:35:59.650923 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:59.650358    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq"]
Oct 30 05:35:59.664972 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:59.661429    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-83e83e-openshift-default-796ccd4c65-4dwvq"]
Oct 30 05:35:59.739303 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:59.739261    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg"]
Oct 30 05:35:59.739638 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:59.739606    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg" podUID="a5620415-5516-441d-ab13-3a574d363b59" containerName="istio-proxy" containerID="cri-o://1fe205a80e215e24adc2495bfd885611e4219226c9476eef1a4178f43f52db7b" gracePeriod=30
Oct 30 05:35:59.877925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:35:59.873926    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:00.226093 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:00.226055    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:00.609738 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:00.609700    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:00.728059 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:00.728019    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:00.898635 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:00.898546    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fecce31e-a6fa-443f-aedf-327ef9710f2a" path="/var/lib/kubelet/pods/fecce31e-a6fa-443f-aedf-327ef9710f2a/volumes"
Oct 30 05:36:00.968950 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:00.968912    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:01.827867 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:01.827807    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:03.322402 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:03.322359    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:04.253883 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:04.253849    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:04.356818 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:04.356790    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:04.480534 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:04.480500    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:05.020192 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:05.020136    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg"
Oct 30 05:36:05.023421 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:05.023390    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5620415-5516-441d-ab13-3a574d363b59-kube-api-access-hfq66\" (UniqueName: \"kubernetes.io/projected/a5620415-5516-441d-ab13-3a574d363b59-kube-api-access-hfq66\") pod \"a5620415-5516-441d-ab13-3a574d363b59\" (UID: \"a5620415-5516-441d-ab13-3a574d363b59\") "
Oct 30 05:36:05.023494 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:05.023451    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5620415-5516-441d-ab13-3a574d363b59-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5620415-5516-441d-ab13-3a574d363b59-workload-certs\") pod \"a5620415-5516-441d-ab13-3a574d363b59\" (UID: \"a5620415-5516-441d-ab13-3a574d363b59\") "
Oct 30 05:36:05.023545 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:05.023509    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5620415-5516-441d-ab13-3a574d363b59-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5620415-5516-441d-ab13-3a574d363b59-credential-socket\") pod \"a5620415-5516-441d-ab13-3a574d363b59\" (UID: \"a5620415-5516-441d-ab13-3a574d363b59\") "
Oct 30 05:36:05.023604 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:05.023582    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5620415-5516-441d-ab13-3a574d363b59-istio-token\" (UniqueName: \"kubernetes.io/projected/a5620415-5516-441d-ab13-3a574d363b59-istio-token\") pod \"a5620415-5516-441d-ab13-3a574d363b59\" (UID: \"a5620415-5516-441d-ab13-3a574d363b59\") "
Oct 30 05:36:05.023659 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:05.023633    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5620415-5516-441d-ab13-3a574d363b59-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5620415-5516-441d-ab13-3a574d363b59-workload-socket\") pod \"a5620415-5516-441d-ab13-3a574d363b59\" (UID: \"a5620415-5516-441d-ab13-3a574d363b59\") "
Oct 30 05:36:05.023712 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:05.023698    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a5620415-5516-441d-ab13-3a574d363b59-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5620415-5516-441d-ab13-3a574d363b59-istio-podinfo\") pod \"a5620415-5516-441d-ab13-3a574d363b59\" (UID: \"a5620415-5516-441d-ab13-3a574d363b59\") "
Oct 30 05:36:05.023771 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:05.023741    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5620415-5516-441d-ab13-3a574d363b59-workload-certs" pod "a5620415-5516-441d-ab13-3a574d363b59" (UID: "a5620415-5516-441d-ab13-3a574d363b59"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:36:05.023771 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:05.023761    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5620415-5516-441d-ab13-3a574d363b59-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5620415-5516-441d-ab13-3a574d363b59-istio-data\") pod \"a5620415-5516-441d-ab13-3a574d363b59\" (UID: \"a5620415-5516-441d-ab13-3a574d363b59\") "
Oct 30 05:36:05.023880 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:05.023838    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5620415-5516-441d-ab13-3a574d363b59-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5620415-5516-441d-ab13-3a574d363b59-istio-envoy\") pod \"a5620415-5516-441d-ab13-3a574d363b59\" (UID: \"a5620415-5516-441d-ab13-3a574d363b59\") "
Oct 30 05:36:05.023880 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:05.023855    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5620415-5516-441d-ab13-3a574d363b59-credential-socket" pod "a5620415-5516-441d-ab13-3a574d363b59" (UID: "a5620415-5516-441d-ab13-3a574d363b59"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:36:05.024048 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:05.023933    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5620415-5516-441d-ab13-3a574d363b59-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5620415-5516-441d-ab13-3a574d363b59-istiod-ca-cert\") pod \"a5620415-5516-441d-ab13-3a574d363b59\" (UID: \"a5620415-5516-441d-ab13-3a574d363b59\") "
Oct 30 05:36:05.024048 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:05.023970    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5620415-5516-441d-ab13-3a574d363b59-workload-socket" pod "a5620415-5516-441d-ab13-3a574d363b59" (UID: "a5620415-5516-441d-ab13-3a574d363b59"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:36:05.024048 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:05.024028    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5620415-5516-441d-ab13-3a574d363b59-istio-data" pod "a5620415-5516-441d-ab13-3a574d363b59" (UID: "a5620415-5516-441d-ab13-3a574d363b59"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:36:05.024367 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:05.024339    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5620415-5516-441d-ab13-3a574d363b59-istiod-ca-cert" pod "a5620415-5516-441d-ab13-3a574d363b59" (UID: "a5620415-5516-441d-ab13-3a574d363b59"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:36:05.024664 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:05.024642    5898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5620415-5516-441d-ab13-3a574d363b59-workload-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:36:05.024755 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:05.024670    5898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5620415-5516-441d-ab13-3a574d363b59-credential-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:36:05.024755 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:05.024687    5898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5620415-5516-441d-ab13-3a574d363b59-workload-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:36:05.024755 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:05.024702    5898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5620415-5516-441d-ab13-3a574d363b59-istio-data\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:36:05.024755 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:05.024718    5898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5620415-5516-441d-ab13-3a574d363b59-istiod-ca-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:36:05.025733 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:05.025712    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5620415-5516-441d-ab13-3a574d363b59-kube-api-access-hfq66" pod "a5620415-5516-441d-ab13-3a574d363b59" (UID: "a5620415-5516-441d-ab13-3a574d363b59"). InnerVolumeSpecName "kube-api-access-hfq66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:36:05.025837 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:05.025772    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5620415-5516-441d-ab13-3a574d363b59-istio-podinfo" pod "a5620415-5516-441d-ab13-3a574d363b59" (UID: "a5620415-5516-441d-ab13-3a574d363b59"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:36:05.026071 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:05.026053    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5620415-5516-441d-ab13-3a574d363b59-istio-envoy" pod "a5620415-5516-441d-ab13-3a574d363b59" (UID: "a5620415-5516-441d-ab13-3a574d363b59"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:36:05.026129 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:05.026068    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5620415-5516-441d-ab13-3a574d363b59-istio-token" pod "a5620415-5516-441d-ab13-3a574d363b59" (UID: "a5620415-5516-441d-ab13-3a574d363b59"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:36:05.125768 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:05.125732    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfq66\" (UniqueName: \"kubernetes.io/projected/a5620415-5516-441d-ab13-3a574d363b59-kube-api-access-hfq66\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:36:05.125768 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:05.125766    5898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5620415-5516-441d-ab13-3a574d363b59-istio-token\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:36:05.126024 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:05.125781    5898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5620415-5516-441d-ab13-3a574d363b59-istio-podinfo\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:36:05.126024 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:05.125793    5898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5620415-5516-441d-ab13-3a574d363b59-istio-envoy\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:36:05.232445 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:05.232410    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9"]
Oct 30 05:36:05.327052 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:05.327018    5898 generic.go:356] "Generic (PLEG): container finished" podID="a5620415-5516-441d-ab13-3a574d363b59" containerID="1fe205a80e215e24adc2495bfd885611e4219226c9476eef1a4178f43f52db7b" exitCode=0
Oct 30 05:36:05.327243 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:05.327065    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg"
Oct 30 05:36:05.327243 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:05.327117    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg" event={"ID":"a5620415-5516-441d-ab13-3a574d363b59","Type":"ContainerDied","Data":"1fe205a80e215e24adc2495bfd885611e4219226c9476eef1a4178f43f52db7b"}
Oct 30 05:36:05.327243 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:05.327180    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg" event={"ID":"a5620415-5516-441d-ab13-3a574d363b59","Type":"ContainerDied","Data":"40d7dc9c41158cb668586000b7cd8c4dc91c6adc7cc086856af975b244d12e8b"}
Oct 30 05:36:05.327243 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:05.327212    5898 scope.go:122] "RemoveContainer" containerID="1fe205a80e215e24adc2495bfd885611e4219226c9476eef1a4178f43f52db7b"
Oct 30 05:36:05.340895 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:05.340873    5898 scope.go:122] "RemoveContainer" containerID="1fe205a80e215e24adc2495bfd885611e4219226c9476eef1a4178f43f52db7b"
Oct 30 05:36:05.341224 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:36:05.341203    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fe205a80e215e24adc2495bfd885611e4219226c9476eef1a4178f43f52db7b\": container with ID starting with 1fe205a80e215e24adc2495bfd885611e4219226c9476eef1a4178f43f52db7b not found: ID does not exist" containerID="1fe205a80e215e24adc2495bfd885611e4219226c9476eef1a4178f43f52db7b"
Oct 30 05:36:05.341335 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:05.341229    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fe205a80e215e24adc2495bfd885611e4219226c9476eef1a4178f43f52db7b"} err="failed to get container status \"1fe205a80e215e24adc2495bfd885611e4219226c9476eef1a4178f43f52db7b\": rpc error: code = NotFound desc = could not find container \"1fe205a80e215e24adc2495bfd885611e4219226c9476eef1a4178f43f52db7b\": container with ID starting with 1fe205a80e215e24adc2495bfd885611e4219226c9476eef1a4178f43f52db7b not found: ID does not exist"
Oct 30 05:36:05.365957 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:05.365923    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:05.391577 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:05.391536    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg"]
Oct 30 05:36:05.396976 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:05.396950    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-674082-openshift-default-577d994559-wshjg"]
Oct 30 05:36:06.574974 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:06.574915    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:06.892094 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:06.892014    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5620415-5516-441d-ab13-3a574d363b59" path="/var/lib/kubelet/pods/a5620415-5516-441d-ab13-3a574d363b59/volumes"
Oct 30 05:36:07.982472 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:07.982411    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:08.269558 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:08.269471    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:09.235321 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:09.235281    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:11.059062 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:11.059030    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:11.192941 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:11.192893    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:13.225067 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:13.225030    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:15.158120 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:15.158081    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:17.035418 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:17.035387    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:18.534437 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:18.534397    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:18.679410 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:18.679378    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:18.831293 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:18.831254    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:19.000165 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:19.000132    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:19.173552 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:19.173478    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:19.518611 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:19.518509    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:19.652348 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:19.652312    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:20.261575 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:20.261528    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9" podUID="1647f909-5580-4219-a2e1-30b4cd994379" containerName="main" containerID="cri-o://70240ed3054547cf492254df2362e4f9c35094c6497d02127da86c58a817074d" gracePeriod=45
Oct 30 05:36:20.324973 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:20.324930    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:20.435065 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:20.435033    5898 generic.go:356] "Generic (PLEG): container finished" podID="1647f909-5580-4219-a2e1-30b4cd994379" containerID="70240ed3054547cf492254df2362e4f9c35094c6497d02127da86c58a817074d" exitCode=0
Oct 30 05:36:20.435244 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:20.435107    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9" event={"ID":"1647f909-5580-4219-a2e1-30b4cd994379","Type":"ContainerDied","Data":"70240ed3054547cf492254df2362e4f9c35094c6497d02127da86c58a817074d"}
Oct 30 05:36:20.532306 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:20.532248    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9"
Oct 30 05:36:20.666216 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:20.666178    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1647f909-5580-4219-a2e1-30b4cd994379-home\" (UniqueName: \"kubernetes.io/empty-dir/1647f909-5580-4219-a2e1-30b4cd994379-home\") pod \"1647f909-5580-4219-a2e1-30b4cd994379\" (UID: \"1647f909-5580-4219-a2e1-30b4cd994379\") "
Oct 30 05:36:20.666711 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:20.666385    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1647f909-5580-4219-a2e1-30b4cd994379-tls-certs\" (UniqueName: \"kubernetes.io/secret/1647f909-5580-4219-a2e1-30b4cd994379-tls-certs\") pod \"1647f909-5580-4219-a2e1-30b4cd994379\" (UID: \"1647f909-5580-4219-a2e1-30b4cd994379\") "
Oct 30 05:36:20.666711 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:20.666447    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1647f909-5580-4219-a2e1-30b4cd994379-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1647f909-5580-4219-a2e1-30b4cd994379-model-cache\") pod \"1647f909-5580-4219-a2e1-30b4cd994379\" (UID: \"1647f909-5580-4219-a2e1-30b4cd994379\") "
Oct 30 05:36:20.666711 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:20.666483    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1647f909-5580-4219-a2e1-30b4cd994379-home" pod "1647f909-5580-4219-a2e1-30b4cd994379" (UID: "1647f909-5580-4219-a2e1-30b4cd994379"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:36:20.666711 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:20.666539    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1647f909-5580-4219-a2e1-30b4cd994379-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1647f909-5580-4219-a2e1-30b4cd994379-dshm\") pod \"1647f909-5580-4219-a2e1-30b4cd994379\" (UID: \"1647f909-5580-4219-a2e1-30b4cd994379\") "
Oct 30 05:36:20.666711 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:20.666626    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1647f909-5580-4219-a2e1-30b4cd994379-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1647f909-5580-4219-a2e1-30b4cd994379-tmp-dir\") pod \"1647f909-5580-4219-a2e1-30b4cd994379\" (UID: \"1647f909-5580-4219-a2e1-30b4cd994379\") "
Oct 30 05:36:20.667040 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:20.666716    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1647f909-5580-4219-a2e1-30b4cd994379-kube-api-access-2vrqp\" (UniqueName: \"kubernetes.io/projected/1647f909-5580-4219-a2e1-30b4cd994379-kube-api-access-2vrqp\") pod \"1647f909-5580-4219-a2e1-30b4cd994379\" (UID: \"1647f909-5580-4219-a2e1-30b4cd994379\") "
Oct 30 05:36:20.667040 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:20.666732    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1647f909-5580-4219-a2e1-30b4cd994379-model-cache" pod "1647f909-5580-4219-a2e1-30b4cd994379" (UID: "1647f909-5580-4219-a2e1-30b4cd994379"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:36:20.667040 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:20.666817    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1647f909-5580-4219-a2e1-30b4cd994379-tmp-dir" pod "1647f909-5580-4219-a2e1-30b4cd994379" (UID: "1647f909-5580-4219-a2e1-30b4cd994379"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:36:20.667340 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:20.667320    5898 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1647f909-5580-4219-a2e1-30b4cd994379-tmp-dir\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:36:20.667413 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:20.667343    5898 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1647f909-5580-4219-a2e1-30b4cd994379-home\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:36:20.667413 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:20.667356    5898 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1647f909-5580-4219-a2e1-30b4cd994379-model-cache\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:36:20.668874 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:20.668847    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1647f909-5580-4219-a2e1-30b4cd994379-tls-certs" pod "1647f909-5580-4219-a2e1-30b4cd994379" (UID: "1647f909-5580-4219-a2e1-30b4cd994379"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:36:20.669040 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:20.668973    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1647f909-5580-4219-a2e1-30b4cd994379-kube-api-access-2vrqp" pod "1647f909-5580-4219-a2e1-30b4cd994379" (UID: "1647f909-5580-4219-a2e1-30b4cd994379"). InnerVolumeSpecName "kube-api-access-2vrqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:36:20.669162 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:20.669139    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1647f909-5580-4219-a2e1-30b4cd994379-dshm" pod "1647f909-5580-4219-a2e1-30b4cd994379" (UID: "1647f909-5580-4219-a2e1-30b4cd994379"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:36:20.768690 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:20.768652    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vrqp\" (UniqueName: \"kubernetes.io/projected/1647f909-5580-4219-a2e1-30b4cd994379-kube-api-access-2vrqp\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:36:20.768690 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:20.768686    5898 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1647f909-5580-4219-a2e1-30b4cd994379-tls-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:36:20.768925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:20.768703    5898 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1647f909-5580-4219-a2e1-30b4cd994379-dshm\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:36:21.443291 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:21.443257    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9" event={"ID":"1647f909-5580-4219-a2e1-30b4cd994379","Type":"ContainerDied","Data":"ecdd6f593bb7feda4576614f45ebe9d16a45c3546348e422117a57bab77ac287"}
Oct 30 05:36:21.443291 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:21.443280    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9"
Oct 30 05:36:21.443722 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:21.443309    5898 scope.go:122] "RemoveContainer" containerID="70240ed3054547cf492254df2362e4f9c35094c6497d02127da86c58a817074d"
Oct 30 05:36:21.506182 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:21.506147    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9"]
Oct 30 05:36:21.516789 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:21.516761    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-d55037/simulator-w4-d55037-kserve-6b959b69bc-cjlv9"]
Oct 30 05:36:21.936734 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:21.936698    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:22.560677 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:22.560641    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:22.892082 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:22.892050    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1647f909-5580-4219-a2e1-30b4cd994379" path="/var/lib/kubelet/pods/1647f909-5580-4219-a2e1-30b4cd994379/volumes"
Oct 30 05:36:23.144975 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:23.140455    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:23.344756 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:23.344720    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:24.535588 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:24.535546    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:26.478753 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:26.478715    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:26.821981 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:36:26.821948    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:36:26.929159 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:26.929114    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:27.309156 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:27.309059    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:27.445168 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:27.445131    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:27.650852 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:27.650818    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:28.697949 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:28.697877    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:28.804753 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:28.804719    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:29.698847 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:29.698811    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:31.693984 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:31.693933    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:33.477813 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:33.477772    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:35.270944 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:35.270883    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:35.693894 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:35.693837    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:36:37.819824 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:37.819789    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:38.987343 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:38.987307    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ff4760-78747f9bc9-hwqpl"]
Oct 30 05:36:39.096116 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:39.096088    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:39.220579 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:39.220542    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:39.802317 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:39.802280    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:41.429520 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:41.429488    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:41.477387 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:41.477347    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-02cffa-858d55bbb-hxc8w"]
Oct 30 05:36:41.477612 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:41.477589    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-02cffa-858d55bbb-hxc8w" podUID="dc5b5678-4d59-41b9-857b-8c502ec7eb28" containerName="payload-processing" containerID="cri-o://c7fa77c07b732e84a6f5c1b7b64a9af9e7909ef29b7e271a7035005e52969e44" gracePeriod=30
Oct 30 05:36:41.509278 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:41.509237    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-02cffa-7875b685ff-pchf5"]
Oct 30 05:36:41.519468 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:41.519384    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02cffa-7875b685ff-pchf5" podUID="4f19c225-09ec-43b9-af12-7c05b7546d72" containerName="payload-pre-processing" containerID="cri-o://5e6eb7c72854600ad6ef9e8dfa0abf57bd140a3ed92a833063a2cd0ae6cd29a3" gracePeriod=30
Oct 30 05:36:41.582895 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:41.579724    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:41.710448 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:41.710363    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:41.723120 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:41.722782    5898 generic.go:356] "Generic (PLEG): container finished" podID="4f19c225-09ec-43b9-af12-7c05b7546d72" containerID="5e6eb7c72854600ad6ef9e8dfa0abf57bd140a3ed92a833063a2cd0ae6cd29a3" exitCode=0
Oct 30 05:36:41.723120 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:41.723071    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02cffa-7875b685ff-pchf5" event={"ID":"4f19c225-09ec-43b9-af12-7c05b7546d72","Type":"ContainerDied","Data":"5e6eb7c72854600ad6ef9e8dfa0abf57bd140a3ed92a833063a2cd0ae6cd29a3"}
Oct 30 05:36:41.725082 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:41.725058    5898 generic.go:356] "Generic (PLEG): container finished" podID="dc5b5678-4d59-41b9-857b-8c502ec7eb28" containerID="c7fa77c07b732e84a6f5c1b7b64a9af9e7909ef29b7e271a7035005e52969e44" exitCode=0
Oct 30 05:36:41.725229 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:41.725121    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-02cffa-858d55bbb-hxc8w" event={"ID":"dc5b5678-4d59-41b9-857b-8c502ec7eb28","Type":"ContainerDied","Data":"c7fa77c07b732e84a6f5c1b7b64a9af9e7909ef29b7e271a7035005e52969e44"}
Oct 30 05:36:41.802588 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:41.802561    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-02cffa-858d55bbb-hxc8w"
Oct 30 05:36:41.824279 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:41.824250    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc5b5678-4d59-41b9-857b-8c502ec7eb28-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc5b5678-4d59-41b9-857b-8c502ec7eb28-plugins-config-volume\") pod \"dc5b5678-4d59-41b9-857b-8c502ec7eb28\" (UID: \"dc5b5678-4d59-41b9-857b-8c502ec7eb28\") "
Oct 30 05:36:41.824458 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:41.824346    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc5b5678-4d59-41b9-857b-8c502ec7eb28-kube-api-access-nm84s\" (UniqueName: \"kubernetes.io/projected/dc5b5678-4d59-41b9-857b-8c502ec7eb28-kube-api-access-nm84s\") pod \"dc5b5678-4d59-41b9-857b-8c502ec7eb28\" (UID: \"dc5b5678-4d59-41b9-857b-8c502ec7eb28\") "
Oct 30 05:36:41.824695 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:41.824668    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc5b5678-4d59-41b9-857b-8c502ec7eb28-plugins-config-volume" pod "dc5b5678-4d59-41b9-857b-8c502ec7eb28" (UID: "dc5b5678-4d59-41b9-857b-8c502ec7eb28"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:36:41.824877 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:41.824861    5898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc5b5678-4d59-41b9-857b-8c502ec7eb28-plugins-config-volume\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:36:41.827074 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:41.827051    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc5b5678-4d59-41b9-857b-8c502ec7eb28-kube-api-access-nm84s" pod "dc5b5678-4d59-41b9-857b-8c502ec7eb28" (UID: "dc5b5678-4d59-41b9-857b-8c502ec7eb28"). InnerVolumeSpecName "kube-api-access-nm84s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:36:41.866608 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:41.866564    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02cffa-7875b685ff-pchf5"
Oct 30 05:36:41.944926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:41.943976    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f19c225-09ec-43b9-af12-7c05b7546d72-kube-api-access-h798n\" (UniqueName: \"kubernetes.io/projected/4f19c225-09ec-43b9-af12-7c05b7546d72-kube-api-access-h798n\") pod \"4f19c225-09ec-43b9-af12-7c05b7546d72\" (UID: \"4f19c225-09ec-43b9-af12-7c05b7546d72\") "
Oct 30 05:36:41.944926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:41.944060    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f19c225-09ec-43b9-af12-7c05b7546d72-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f19c225-09ec-43b9-af12-7c05b7546d72-plugins-config-volume\") pod \"4f19c225-09ec-43b9-af12-7c05b7546d72\" (UID: \"4f19c225-09ec-43b9-af12-7c05b7546d72\") "
Oct 30 05:36:41.944926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:41.944867    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm84s\" (UniqueName: \"kubernetes.io/projected/dc5b5678-4d59-41b9-857b-8c502ec7eb28-kube-api-access-nm84s\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:36:41.959340 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:41.955137    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f19c225-09ec-43b9-af12-7c05b7546d72-plugins-config-volume" pod "4f19c225-09ec-43b9-af12-7c05b7546d72" (UID: "4f19c225-09ec-43b9-af12-7c05b7546d72"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:36:42.029513 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:42.029203    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:42.046850 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:42.046742    5898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f19c225-09ec-43b9-af12-7c05b7546d72-plugins-config-volume\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:36:42.098036 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:42.097979    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f19c225-09ec-43b9-af12-7c05b7546d72-kube-api-access-h798n" pod "4f19c225-09ec-43b9-af12-7c05b7546d72" (UID: "4f19c225-09ec-43b9-af12-7c05b7546d72"). InnerVolumeSpecName "kube-api-access-h798n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:36:42.148924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:42.148646    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h798n\" (UniqueName: \"kubernetes.io/projected/4f19c225-09ec-43b9-af12-7c05b7546d72-kube-api-access-h798n\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:36:42.219371 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:42.219329    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:42.381915 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:42.381870    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:42.732798 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:42.732716    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-02cffa-858d55bbb-hxc8w"
Oct 30 05:36:42.732798 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:42.732754    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-02cffa-858d55bbb-hxc8w" event={"ID":"dc5b5678-4d59-41b9-857b-8c502ec7eb28","Type":"ContainerDied","Data":"3ee3500e6f2839b983f06bd6096212b64c8eaf4d4588d88c017568efae1d3e46"}
Oct 30 05:36:42.733357 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:42.732829    5898 scope.go:122] "RemoveContainer" containerID="c7fa77c07b732e84a6f5c1b7b64a9af9e7909ef29b7e271a7035005e52969e44"
Oct 30 05:36:42.734588 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:42.734386    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02cffa-7875b685ff-pchf5"
Oct 30 05:36:42.734588 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:42.734386    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02cffa-7875b685ff-pchf5" event={"ID":"4f19c225-09ec-43b9-af12-7c05b7546d72","Type":"ContainerDied","Data":"661ad3ee3c4c3d0661cc08dfcb3d002e65e8bf6fc3a8fd2475be933c8f4fd3d4"}
Oct 30 05:36:42.747802 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:42.747636    5898 scope.go:122] "RemoveContainer" containerID="5e6eb7c72854600ad6ef9e8dfa0abf57bd140a3ed92a833063a2cd0ae6cd29a3"
Oct 30 05:36:42.758986 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:42.758951    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:42.799248 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:42.799216    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-02cffa-858d55bbb-hxc8w"]
Oct 30 05:36:42.812209 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:42.812179    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-02cffa-858d55bbb-hxc8w"]
Oct 30 05:36:42.848014 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:42.847980    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-02cffa-7875b685ff-pchf5"]
Oct 30 05:36:42.852393 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:42.852363    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-02cffa-7875b685ff-pchf5"]
Oct 30 05:36:42.893132 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:42.893098    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f19c225-09ec-43b9-af12-7c05b7546d72" path="/var/lib/kubelet/pods/4f19c225-09ec-43b9-af12-7c05b7546d72/volumes"
Oct 30 05:36:42.893548 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:42.893532    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc5b5678-4d59-41b9-857b-8c502ec7eb28" path="/var/lib/kubelet/pods/dc5b5678-4d59-41b9-857b-8c502ec7eb28/volumes"
Oct 30 05:36:42.901237 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:42.901207    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:43.803721 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:43.803685    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:43.987980 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:43.987888    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ff4760-78747f9bc9-hwqpl" podUID="7063ee78-5b1b-4f4a-b1f0-7898af800db1" containerName="payload-pre-processing" containerID="cri-o://e9868171209a0cafc17670b186989bc32023e1ea90a6b4c06f133f3fa9960bfa" gracePeriod=25
Oct 30 05:36:45.921217 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:45.921180    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:46.072174 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:46.072139    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:46.242617 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:46.242501    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:47.542146 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:47.542110    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:49.623694 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:49.623643    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:50.160778 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:50.160734    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:50.300978 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:50.300946    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:50.597652 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:50.597615    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k"]
Oct 30 05:36:50.598143 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:50.598107    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k" podUID="4c2b56cf-6fbe-4787-86db-1692ecbb41e0" containerName="istio-proxy" containerID="cri-o://6d5251f6de12748dd21f851055ebf9a520f34f33a590f51060356d0594dc8a64" gracePeriod=30
Oct 30 05:36:50.618571 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:50.618540    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw"]
Oct 30 05:36:50.618797 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:50.618769    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw" podUID="268c3238-4e5e-4907-b5c0-cf47164701ea" containerName="istio-proxy" containerID="cri-o://18fece5dd1ef979c4b12e33f0b095af6f204fe0d53079458da9f7ca662fb76f4" gracePeriod=30
Oct 30 05:36:50.629644 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:50.629619    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:51.838784 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:51.838746    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:51.945372 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:51.945343    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:52.063142 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:52.063103    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:53.048195 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:53.048103    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ff4760-78747f9bc9-hwqpl" podUID="7063ee78-5b1b-4f4a-b1f0-7898af800db1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:36:54.075967 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:54.075920    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:54.192679 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:54.192641    5898 patch_prober.go:29] interesting pod/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 05:36:54.192852 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:54.192710    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw" podUID="268c3238-4e5e-4907-b5c0-cf47164701ea" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 05:36:55.905757 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:55.905732    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k"
Oct 30 05:36:55.928952 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:55.928918    5898 generic.go:356] "Generic (PLEG): container finished" podID="4c2b56cf-6fbe-4787-86db-1692ecbb41e0" containerID="6d5251f6de12748dd21f851055ebf9a520f34f33a590f51060356d0594dc8a64" exitCode=0
Oct 30 05:36:55.929140 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:55.928974    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k"
Oct 30 05:36:55.929140 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:55.929126    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k" event={"ID":"4c2b56cf-6fbe-4787-86db-1692ecbb41e0","Type":"ContainerDied","Data":"6d5251f6de12748dd21f851055ebf9a520f34f33a590f51060356d0594dc8a64"}
Oct 30 05:36:55.929299 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:55.929181    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k" event={"ID":"4c2b56cf-6fbe-4787-86db-1692ecbb41e0","Type":"ContainerDied","Data":"a0648b8bff3d67b24a885fd6fd55c82be078bcd5d36094edcaa8c404ebdf2aad"}
Oct 30 05:36:55.929299 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:55.929212    5898 scope.go:122] "RemoveContainer" containerID="6d5251f6de12748dd21f851055ebf9a520f34f33a590f51060356d0594dc8a64"
Oct 30 05:36:55.931129 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:55.931100    5898 generic.go:356] "Generic (PLEG): container finished" podID="268c3238-4e5e-4907-b5c0-cf47164701ea" containerID="18fece5dd1ef979c4b12e33f0b095af6f204fe0d53079458da9f7ca662fb76f4" exitCode=0
Oct 30 05:36:55.931238 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:55.931168    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw" event={"ID":"268c3238-4e5e-4907-b5c0-cf47164701ea","Type":"ContainerDied","Data":"18fece5dd1ef979c4b12e33f0b095af6f204fe0d53079458da9f7ca662fb76f4"}
Oct 30 05:36:55.949388 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:55.949370    5898 scope.go:122] "RemoveContainer" containerID="6d5251f6de12748dd21f851055ebf9a520f34f33a590f51060356d0594dc8a64"
Oct 30 05:36:55.949687 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:36:55.949665    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d5251f6de12748dd21f851055ebf9a520f34f33a590f51060356d0594dc8a64\": container with ID starting with 6d5251f6de12748dd21f851055ebf9a520f34f33a590f51060356d0594dc8a64 not found: ID does not exist" containerID="6d5251f6de12748dd21f851055ebf9a520f34f33a590f51060356d0594dc8a64"
Oct 30 05:36:55.949792 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:55.949694    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d5251f6de12748dd21f851055ebf9a520f34f33a590f51060356d0594dc8a64"} err="failed to get container status \"6d5251f6de12748dd21f851055ebf9a520f34f33a590f51060356d0594dc8a64\": rpc error: code = NotFound desc = could not find container \"6d5251f6de12748dd21f851055ebf9a520f34f33a590f51060356d0594dc8a64\": container with ID starting with 6d5251f6de12748dd21f851055ebf9a520f34f33a590f51060356d0594dc8a64 not found: ID does not exist"
Oct 30 05:36:55.952311 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:55.952288    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw"
Oct 30 05:36:56.017287 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:56.017247    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c2b56cf-6fbe-4787-86db-1692ecbb41e0-istio-token\" (UniqueName: \"kubernetes.io/projected/4c2b56cf-6fbe-4787-86db-1692ecbb41e0-istio-token\") pod \"4c2b56cf-6fbe-4787-86db-1692ecbb41e0\" (UID: \"4c2b56cf-6fbe-4787-86db-1692ecbb41e0\") "
Oct 30 05:36:56.017503 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:56.017305    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c2b56cf-6fbe-4787-86db-1692ecbb41e0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c2b56cf-6fbe-4787-86db-1692ecbb41e0-istiod-ca-cert\") pod \"4c2b56cf-6fbe-4787-86db-1692ecbb41e0\" (UID: \"4c2b56cf-6fbe-4787-86db-1692ecbb41e0\") "
Oct 30 05:36:56.017503 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:56.017384    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c2b56cf-6fbe-4787-86db-1692ecbb41e0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c2b56cf-6fbe-4787-86db-1692ecbb41e0-istio-envoy\") pod \"4c2b56cf-6fbe-4787-86db-1692ecbb41e0\" (UID: \"4c2b56cf-6fbe-4787-86db-1692ecbb41e0\") "
Oct 30 05:36:56.017503 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:56.017432    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c2b56cf-6fbe-4787-86db-1692ecbb41e0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c2b56cf-6fbe-4787-86db-1692ecbb41e0-workload-socket\") pod \"4c2b56cf-6fbe-4787-86db-1692ecbb41e0\" (UID: \"4c2b56cf-6fbe-4787-86db-1692ecbb41e0\") "
Oct 30 05:36:56.017713 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:56.017506    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c2b56cf-6fbe-4787-86db-1692ecbb41e0-kube-api-access-pwjlb\" (UniqueName: \"kubernetes.io/projected/4c2b56cf-6fbe-4787-86db-1692ecbb41e0-kube-api-access-pwjlb\") pod \"4c2b56cf-6fbe-4787-86db-1692ecbb41e0\" (UID: \"4c2b56cf-6fbe-4787-86db-1692ecbb41e0\") "
Oct 30 05:36:56.017713 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:56.017542    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c2b56cf-6fbe-4787-86db-1692ecbb41e0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c2b56cf-6fbe-4787-86db-1692ecbb41e0-credential-socket\") pod \"4c2b56cf-6fbe-4787-86db-1692ecbb41e0\" (UID: \"4c2b56cf-6fbe-4787-86db-1692ecbb41e0\") "
Oct 30 05:36:56.017713 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:56.017566    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c2b56cf-6fbe-4787-86db-1692ecbb41e0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c2b56cf-6fbe-4787-86db-1692ecbb41e0-istio-data\") pod \"4c2b56cf-6fbe-4787-86db-1692ecbb41e0\" (UID: \"4c2b56cf-6fbe-4787-86db-1692ecbb41e0\") "
Oct 30 05:36:56.017713 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:56.017600    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c2b56cf-6fbe-4787-86db-1692ecbb41e0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c2b56cf-6fbe-4787-86db-1692ecbb41e0-workload-certs\") pod \"4c2b56cf-6fbe-4787-86db-1692ecbb41e0\" (UID: \"4c2b56cf-6fbe-4787-86db-1692ecbb41e0\") "
Oct 30 05:36:56.017713 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:56.017647    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c2b56cf-6fbe-4787-86db-1692ecbb41e0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c2b56cf-6fbe-4787-86db-1692ecbb41e0-istio-podinfo\") pod \"4c2b56cf-6fbe-4787-86db-1692ecbb41e0\" (UID: \"4c2b56cf-6fbe-4787-86db-1692ecbb41e0\") "
Oct 30 05:36:56.018879 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:56.018851    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c2b56cf-6fbe-4787-86db-1692ecbb41e0-istiod-ca-cert" pod "4c2b56cf-6fbe-4787-86db-1692ecbb41e0" (UID: "4c2b56cf-6fbe-4787-86db-1692ecbb41e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:36:56.019170 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:56.019144    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c2b56cf-6fbe-4787-86db-1692ecbb41e0-workload-certs" pod "4c2b56cf-6fbe-4787-86db-1692ecbb41e0" (UID: "4c2b56cf-6fbe-4787-86db-1692ecbb41e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:36:56.019170 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:56.019160    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c2b56cf-6fbe-4787-86db-1692ecbb41e0-istio-data" pod "4c2b56cf-6fbe-4787-86db-1692ecbb41e0" (UID: "4c2b56cf-6fbe-4787-86db-1692ecbb41e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:36:56.019339 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:56.019299    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c2b56cf-6fbe-4787-86db-1692ecbb41e0-credential-socket" pod "4c2b56cf-6fbe-4787-86db-1692ecbb41e0" (UID: "4c2b56cf-6fbe-4787-86db-1692ecbb41e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:36:56.019523 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:56.019499    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c2b56cf-6fbe-4787-86db-1692ecbb41e0-workload-socket" pod "4c2b56cf-6fbe-4787-86db-1692ecbb41e0" (UID: "4c2b56cf-6fbe-4787-86db-1692ecbb41e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:36:56.021576 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:56.021550    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c2b56cf-6fbe-4787-86db-1692ecbb41e0-istio-podinfo" pod "4c2b56cf-6fbe-4787-86db-1692ecbb41e0" (UID: "4c2b56cf-6fbe-4787-86db-1692ecbb41e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:36:56.021839 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:56.021817    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c2b56cf-6fbe-4787-86db-1692ecbb41e0-istio-token" pod "4c2b56cf-6fbe-4787-86db-1692ecbb41e0" (UID: "4c2b56cf-6fbe-4787-86db-1692ecbb41e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:36:56.021963 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:56.021885    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c2b56cf-6fbe-4787-86db-1692ecbb41e0-istio-envoy" pod "4c2b56cf-6fbe-4787-86db-1692ecbb41e0" (UID: "4c2b56cf-6fbe-4787-86db-1692ecbb41e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:36:56.021963 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:56.021947    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c2b56cf-6fbe-4787-86db-1692ecbb41e0-kube-api-access-pwjlb" pod "4c2b56cf-6fbe-4787-86db-1692ecbb41e0" (UID: "4c2b56cf-6fbe-4787-86db-1692ecbb41e0"). InnerVolumeSpecName "kube-api-access-pwjlb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:36:56.053737 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:56.053699    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:56.124426 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:56.122889    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/268c3238-4e5e-4907-b5c0-cf47164701ea-istio-token\" (UniqueName: \"kubernetes.io/projected/268c3238-4e5e-4907-b5c0-cf47164701ea-istio-token\") pod \"268c3238-4e5e-4907-b5c0-cf47164701ea\" (UID: \"268c3238-4e5e-4907-b5c0-cf47164701ea\") "
Oct 30 05:36:56.124792 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:56.124769    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/268c3238-4e5e-4907-b5c0-cf47164701ea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/268c3238-4e5e-4907-b5c0-cf47164701ea-istio-data\") pod \"268c3238-4e5e-4907-b5c0-cf47164701ea\" (UID: \"268c3238-4e5e-4907-b5c0-cf47164701ea\") "
Oct 30 05:36:56.124955 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:56.124941    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/268c3238-4e5e-4907-b5c0-cf47164701ea-kube-api-access-pvhzz\" (UniqueName: \"kubernetes.io/projected/268c3238-4e5e-4907-b5c0-cf47164701ea-kube-api-access-pvhzz\") pod \"268c3238-4e5e-4907-b5c0-cf47164701ea\" (UID: \"268c3238-4e5e-4907-b5c0-cf47164701ea\") "
Oct 30 05:36:56.125101 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:56.125073    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/268c3238-4e5e-4907-b5c0-cf47164701ea-istio-data" pod "268c3238-4e5e-4907-b5c0-cf47164701ea" (UID: "268c3238-4e5e-4907-b5c0-cf47164701ea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:36:56.125204 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:56.125190    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/268c3238-4e5e-4907-b5c0-cf47164701ea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/268c3238-4e5e-4907-b5c0-cf47164701ea-istiod-ca-cert\") pod \"268c3238-4e5e-4907-b5c0-cf47164701ea\" (UID: \"268c3238-4e5e-4907-b5c0-cf47164701ea\") "
Oct 30 05:36:56.125352 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:56.125335    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/268c3238-4e5e-4907-b5c0-cf47164701ea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/268c3238-4e5e-4907-b5c0-cf47164701ea-workload-socket\") pod \"268c3238-4e5e-4907-b5c0-cf47164701ea\" (UID: \"268c3238-4e5e-4907-b5c0-cf47164701ea\") "
Oct 30 05:36:56.125527 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:56.125513    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/268c3238-4e5e-4907-b5c0-cf47164701ea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/268c3238-4e5e-4907-b5c0-cf47164701ea-istio-envoy\") pod \"268c3238-4e5e-4907-b5c0-cf47164701ea\" (UID: \"268c3238-4e5e-4907-b5c0-cf47164701ea\") "
Oct 30 05:36:56.125642 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:56.125618    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/268c3238-4e5e-4907-b5c0-cf47164701ea-istiod-ca-cert" pod "268c3238-4e5e-4907-b5c0-cf47164701ea" (UID: "268c3238-4e5e-4907-b5c0-cf47164701ea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:36:56.125731 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:56.125718    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/268c3238-4e5e-4907-b5c0-cf47164701ea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/268c3238-4e5e-4907-b5c0-cf47164701ea-istio-podinfo\") pod \"268c3238-4e5e-4907-b5c0-cf47164701ea\" (UID: \"268c3238-4e5e-4907-b5c0-cf47164701ea\") "
Oct 30 05:36:56.125843 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:56.125828    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/268c3238-4e5e-4907-b5c0-cf47164701ea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/268c3238-4e5e-4907-b5c0-cf47164701ea-workload-certs\") pod \"268c3238-4e5e-4907-b5c0-cf47164701ea\" (UID: \"268c3238-4e5e-4907-b5c0-cf47164701ea\") "
Oct 30 05:36:56.125999 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:56.125983    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/268c3238-4e5e-4907-b5c0-cf47164701ea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/268c3238-4e5e-4907-b5c0-cf47164701ea-credential-socket\") pod \"268c3238-4e5e-4907-b5c0-cf47164701ea\" (UID: \"268c3238-4e5e-4907-b5c0-cf47164701ea\") "
Oct 30 05:36:56.126357 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:56.125882    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/268c3238-4e5e-4907-b5c0-cf47164701ea-workload-socket" pod "268c3238-4e5e-4907-b5c0-cf47164701ea" (UID: "268c3238-4e5e-4907-b5c0-cf47164701ea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:36:56.126456 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:56.126327    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/268c3238-4e5e-4907-b5c0-cf47164701ea-credential-socket" pod "268c3238-4e5e-4907-b5c0-cf47164701ea" (UID: "268c3238-4e5e-4907-b5c0-cf47164701ea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:36:56.126620 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:56.126580    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/268c3238-4e5e-4907-b5c0-cf47164701ea-workload-certs" pod "268c3238-4e5e-4907-b5c0-cf47164701ea" (UID: "268c3238-4e5e-4907-b5c0-cf47164701ea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:36:56.127147 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:56.127124    5898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/268c3238-4e5e-4907-b5c0-cf47164701ea-credential-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:36:56.127276 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:56.127259    5898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c2b56cf-6fbe-4787-86db-1692ecbb41e0-istio-token\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:36:56.127384 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:56.127371    5898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c2b56cf-6fbe-4787-86db-1692ecbb41e0-istiod-ca-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:36:56.127504 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:56.127491    5898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c2b56cf-6fbe-4787-86db-1692ecbb41e0-istio-envoy\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:36:56.127655 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:56.127640    5898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/268c3238-4e5e-4907-b5c0-cf47164701ea-istio-data\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:36:56.127794 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:56.127781    5898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c2b56cf-6fbe-4787-86db-1692ecbb41e0-workload-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:36:56.128072 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:56.127175    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/268c3238-4e5e-4907-b5c0-cf47164701ea-istio-token" pod "268c3238-4e5e-4907-b5c0-cf47164701ea" (UID: "268c3238-4e5e-4907-b5c0-cf47164701ea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:36:56.128072 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:56.128038    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/268c3238-4e5e-4907-b5c0-cf47164701ea-istio-podinfo" pod "268c3238-4e5e-4907-b5c0-cf47164701ea" (UID: "268c3238-4e5e-4907-b5c0-cf47164701ea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:36:56.128283 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:56.128267    5898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/268c3238-4e5e-4907-b5c0-cf47164701ea-istiod-ca-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:36:56.128391 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:56.128378    5898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/268c3238-4e5e-4907-b5c0-cf47164701ea-workload-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:36:56.128493 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:56.128479    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwjlb\" (UniqueName: \"kubernetes.io/projected/4c2b56cf-6fbe-4787-86db-1692ecbb41e0-kube-api-access-pwjlb\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:36:56.128593 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:56.128580    5898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c2b56cf-6fbe-4787-86db-1692ecbb41e0-credential-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:36:56.128694 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:56.128682    5898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c2b56cf-6fbe-4787-86db-1692ecbb41e0-istio-data\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:36:56.128806 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:56.128794    5898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c2b56cf-6fbe-4787-86db-1692ecbb41e0-workload-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:36:56.128929 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:56.128915    5898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/268c3238-4e5e-4907-b5c0-cf47164701ea-workload-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:36:56.129292 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:56.129273    5898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c2b56cf-6fbe-4787-86db-1692ecbb41e0-istio-podinfo\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:36:56.129672 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:56.129580    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/268c3238-4e5e-4907-b5c0-cf47164701ea-istio-envoy" pod "268c3238-4e5e-4907-b5c0-cf47164701ea" (UID: "268c3238-4e5e-4907-b5c0-cf47164701ea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:36:56.131170 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:56.131072    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/268c3238-4e5e-4907-b5c0-cf47164701ea-kube-api-access-pvhzz" pod "268c3238-4e5e-4907-b5c0-cf47164701ea" (UID: "268c3238-4e5e-4907-b5c0-cf47164701ea"). InnerVolumeSpecName "kube-api-access-pvhzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:36:56.230864 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:56.230824    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvhzz\" (UniqueName: \"kubernetes.io/projected/268c3238-4e5e-4907-b5c0-cf47164701ea-kube-api-access-pvhzz\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:36:56.230864 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:56.230863    5898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/268c3238-4e5e-4907-b5c0-cf47164701ea-istio-envoy\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:36:56.231154 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:56.230877    5898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/268c3238-4e5e-4907-b5c0-cf47164701ea-istio-podinfo\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:36:56.231154 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:56.230889    5898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/268c3238-4e5e-4907-b5c0-cf47164701ea-istio-token\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:36:56.280367 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:56.280337    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k"]
Oct 30 05:36:56.291302 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:56.291277    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-bfw9k"]
Oct 30 05:36:56.818777 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:36:56.818744    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:36:56.890176 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:56.890095    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c2b56cf-6fbe-4787-86db-1692ecbb41e0" path="/var/lib/kubelet/pods/4c2b56cf-6fbe-4787-86db-1692ecbb41e0/volumes"
Oct 30 05:36:56.939833 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:56.939787    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw" event={"ID":"268c3238-4e5e-4907-b5c0-cf47164701ea","Type":"ContainerDied","Data":"23bbc9e7d19c021592534be6b6084770e666975402f759e97d5cf43b1b8cb7c0"}
Oct 30 05:36:56.939833 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:56.939830    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw"
Oct 30 05:36:56.940319 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:56.939868    5898 scope.go:122] "RemoveContainer" containerID="18fece5dd1ef979c4b12e33f0b095af6f204fe0d53079458da9f7ca662fb76f4"
Oct 30 05:36:57.003943 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:57.003889    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw"]
Oct 30 05:36:57.007734 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:57.007709    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-02cffa-openshift-default-6644dbdd6c-vnkkw"]
Oct 30 05:36:57.879225 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:57.879191    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:36:58.891164 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:58.891130    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="268c3238-4e5e-4907-b5c0-cf47164701ea" path="/var/lib/kubelet/pods/268c3238-4e5e-4907-b5c0-cf47164701ea/volumes"
Oct 30 05:36:59.603210 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:36:59.603170    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:01.144810 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:01.144772    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:01.520995 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:01.519169    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ff4760-7d4b656bc8-55s9p"]
Oct 30 05:37:01.520995 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:01.519412    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ff4760-7d4b656bc8-55s9p" podUID="dfd4e937-c236-4f33-8477-0c55aa536d30" containerName="maas-api" containerID="cri-o://ad814148294ed35b05da2c30180d82dc46f2e3f477f1c83eb1b9a731fb40f256" gracePeriod=30
Oct 30 05:37:01.576908 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:01.576541    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:01.616853 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:37:01.616821    5898 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfd4e937_c236_4f33_8477_0c55aa536d30.slice/crio-ad814148294ed35b05da2c30180d82dc46f2e3f477f1c83eb1b9a731fb40f256.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 05:37:01.699919 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:01.699709    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:01.833224 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:01.833195    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ff4760-7d4b656bc8-55s9p"
Oct 30 05:37:01.847916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:01.847875    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:01.926780 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:01.926750    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfd4e937-c236-4f33-8477-0c55aa536d30-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfd4e937-c236-4f33-8477-0c55aa536d30-openshift-service-ca\") pod \"dfd4e937-c236-4f33-8477-0c55aa536d30\" (UID: \"dfd4e937-c236-4f33-8477-0c55aa536d30\") "
Oct 30 05:37:01.927010 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:01.926813    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfd4e937-c236-4f33-8477-0c55aa536d30-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfd4e937-c236-4f33-8477-0c55aa536d30-metrics-certs\") pod \"dfd4e937-c236-4f33-8477-0c55aa536d30\" (UID: \"dfd4e937-c236-4f33-8477-0c55aa536d30\") "
Oct 30 05:37:01.927113 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:01.927082    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfd4e937-c236-4f33-8477-0c55aa536d30-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfd4e937-c236-4f33-8477-0c55aa536d30-maas-api-tls\") pod \"dfd4e937-c236-4f33-8477-0c55aa536d30\" (UID: \"dfd4e937-c236-4f33-8477-0c55aa536d30\") "
Oct 30 05:37:01.927182 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:01.927161    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfd4e937-c236-4f33-8477-0c55aa536d30-kube-api-access-vmj99\" (UniqueName: \"kubernetes.io/projected/dfd4e937-c236-4f33-8477-0c55aa536d30-kube-api-access-vmj99\") pod \"dfd4e937-c236-4f33-8477-0c55aa536d30\" (UID: \"dfd4e937-c236-4f33-8477-0c55aa536d30\") "
Oct 30 05:37:01.927264 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:01.927237    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfd4e937-c236-4f33-8477-0c55aa536d30-openshift-service-ca" pod "dfd4e937-c236-4f33-8477-0c55aa536d30" (UID: "dfd4e937-c236-4f33-8477-0c55aa536d30"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:37:01.929311 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:01.928686    5898 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfd4e937-c236-4f33-8477-0c55aa536d30-openshift-service-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:37:01.929790 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:01.929768    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfd4e937-c236-4f33-8477-0c55aa536d30-maas-api-tls" pod "dfd4e937-c236-4f33-8477-0c55aa536d30" (UID: "dfd4e937-c236-4f33-8477-0c55aa536d30"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:37:01.930026 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:01.930001    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfd4e937-c236-4f33-8477-0c55aa536d30-metrics-certs" pod "dfd4e937-c236-4f33-8477-0c55aa536d30" (UID: "dfd4e937-c236-4f33-8477-0c55aa536d30"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:37:01.930596 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:01.930576    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfd4e937-c236-4f33-8477-0c55aa536d30-kube-api-access-vmj99" pod "dfd4e937-c236-4f33-8477-0c55aa536d30" (UID: "dfd4e937-c236-4f33-8477-0c55aa536d30"). InnerVolumeSpecName "kube-api-access-vmj99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:37:02.049400 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:02.049232    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfd4e937-c236-4f33-8477-0c55aa536d30-maas-api-tls\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:37:02.049400 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:02.049277    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmj99\" (UniqueName: \"kubernetes.io/projected/dfd4e937-c236-4f33-8477-0c55aa536d30-kube-api-access-vmj99\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:37:02.049400 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:02.049294    5898 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfd4e937-c236-4f33-8477-0c55aa536d30-metrics-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:37:02.094967 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:02.084046    5898 generic.go:356] "Generic (PLEG): container finished" podID="dfd4e937-c236-4f33-8477-0c55aa536d30" containerID="ad814148294ed35b05da2c30180d82dc46f2e3f477f1c83eb1b9a731fb40f256" exitCode=0
Oct 30 05:37:02.094967 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:02.084214    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ff4760-7d4b656bc8-55s9p" event={"ID":"dfd4e937-c236-4f33-8477-0c55aa536d30","Type":"ContainerDied","Data":"ad814148294ed35b05da2c30180d82dc46f2e3f477f1c83eb1b9a731fb40f256"}
Oct 30 05:37:02.094967 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:02.084263    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ff4760-7d4b656bc8-55s9p" event={"ID":"dfd4e937-c236-4f33-8477-0c55aa536d30","Type":"ContainerDied","Data":"39f20a20e2c739b5fa668f1625007c73bb659ed87e21d789bb34ea864f5a8183"}
Oct 30 05:37:02.094967 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:02.084298    5898 scope.go:122] "RemoveContainer" containerID="ad814148294ed35b05da2c30180d82dc46f2e3f477f1c83eb1b9a731fb40f256"
Oct 30 05:37:02.094967 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:02.084460    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ff4760-7d4b656bc8-55s9p"
Oct 30 05:37:02.161381 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:02.160606    5898 scope.go:122] "RemoveContainer" containerID="ad814148294ed35b05da2c30180d82dc46f2e3f477f1c83eb1b9a731fb40f256"
Oct 30 05:37:02.162024 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:37:02.161924    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad814148294ed35b05da2c30180d82dc46f2e3f477f1c83eb1b9a731fb40f256\": container with ID starting with ad814148294ed35b05da2c30180d82dc46f2e3f477f1c83eb1b9a731fb40f256 not found: ID does not exist" containerID="ad814148294ed35b05da2c30180d82dc46f2e3f477f1c83eb1b9a731fb40f256"
Oct 30 05:37:02.162024 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:02.161965    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad814148294ed35b05da2c30180d82dc46f2e3f477f1c83eb1b9a731fb40f256"} err="failed to get container status \"ad814148294ed35b05da2c30180d82dc46f2e3f477f1c83eb1b9a731fb40f256\": rpc error: code = NotFound desc = could not find container \"ad814148294ed35b05da2c30180d82dc46f2e3f477f1c83eb1b9a731fb40f256\": container with ID starting with ad814148294ed35b05da2c30180d82dc46f2e3f477f1c83eb1b9a731fb40f256 not found: ID does not exist"
Oct 30 05:37:02.203332 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:02.203280    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ff4760-7d4b656bc8-55s9p"]
Oct 30 05:37:02.208686 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:02.208634    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ff4760-7d4b656bc8-55s9p"]
Oct 30 05:37:02.324775 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:02.324748    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:02.433196 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:02.433159    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:02.536618 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:02.536586    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:02.653072 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:02.651499    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:02.892043 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:02.892010    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfd4e937-c236-4f33-8477-0c55aa536d30" path="/var/lib/kubelet/pods/dfd4e937-c236-4f33-8477-0c55aa536d30/volumes"
Oct 30 05:37:02.970920 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:02.970818    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:03.047866 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:03.047829    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ff4760-78747f9bc9-hwqpl" podUID="7063ee78-5b1b-4f4a-b1f0-7898af800db1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:37:04.135343 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:04.131637    5898 generic.go:356] "Generic (PLEG): container finished" podID="7063ee78-5b1b-4f4a-b1f0-7898af800db1" containerID="e9868171209a0cafc17670b186989bc32023e1ea90a6b4c06f133f3fa9960bfa" exitCode=0
Oct 30 05:37:04.135343 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:04.131922    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ff4760-78747f9bc9-hwqpl" event={"ID":"7063ee78-5b1b-4f4a-b1f0-7898af800db1","Type":"ContainerDied","Data":"e9868171209a0cafc17670b186989bc32023e1ea90a6b4c06f133f3fa9960bfa"}
Oct 30 05:37:04.316303 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:04.316273    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ff4760-78747f9bc9-hwqpl"
Oct 30 05:37:04.398544 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:04.398512    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7063ee78-5b1b-4f4a-b1f0-7898af800db1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7063ee78-5b1b-4f4a-b1f0-7898af800db1-plugins-config-volume\") pod \"7063ee78-5b1b-4f4a-b1f0-7898af800db1\" (UID: \"7063ee78-5b1b-4f4a-b1f0-7898af800db1\") "
Oct 30 05:37:04.398783 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:04.398755    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7063ee78-5b1b-4f4a-b1f0-7898af800db1-kube-api-access-kkfq2\" (UniqueName: \"kubernetes.io/projected/7063ee78-5b1b-4f4a-b1f0-7898af800db1-kube-api-access-kkfq2\") pod \"7063ee78-5b1b-4f4a-b1f0-7898af800db1\" (UID: \"7063ee78-5b1b-4f4a-b1f0-7898af800db1\") "
Oct 30 05:37:04.398879 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:04.398856    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7063ee78-5b1b-4f4a-b1f0-7898af800db1-plugins-config-volume" pod "7063ee78-5b1b-4f4a-b1f0-7898af800db1" (UID: "7063ee78-5b1b-4f4a-b1f0-7898af800db1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:37:04.399231 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:04.399215    5898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7063ee78-5b1b-4f4a-b1f0-7898af800db1-plugins-config-volume\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:37:04.400914 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:04.400884    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7063ee78-5b1b-4f4a-b1f0-7898af800db1-kube-api-access-kkfq2" pod "7063ee78-5b1b-4f4a-b1f0-7898af800db1" (UID: "7063ee78-5b1b-4f4a-b1f0-7898af800db1"). InnerVolumeSpecName "kube-api-access-kkfq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:37:04.500667 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:04.500636    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkfq2\" (UniqueName: \"kubernetes.io/projected/7063ee78-5b1b-4f4a-b1f0-7898af800db1-kube-api-access-kkfq2\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:37:04.709830 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:04.709795    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:05.139781 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:05.139747    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ff4760-78747f9bc9-hwqpl" event={"ID":"7063ee78-5b1b-4f4a-b1f0-7898af800db1","Type":"ContainerDied","Data":"23ecb3dc2afb2a47c0b4f1504fae3f5628c2179aa77c843c5138a388f15ce34c"}
Oct 30 05:37:05.139781 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:05.139777    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ff4760-78747f9bc9-hwqpl"
Oct 30 05:37:05.140316 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:05.139802    5898 scope.go:122] "RemoveContainer" containerID="e9868171209a0cafc17670b186989bc32023e1ea90a6b4c06f133f3fa9960bfa"
Oct 30 05:37:05.192478 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:05.192442    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ff4760-78747f9bc9-hwqpl"]
Oct 30 05:37:05.201730 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:05.201698    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ff4760-78747f9bc9-hwqpl"]
Oct 30 05:37:06.890859 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:06.890822    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7063ee78-5b1b-4f4a-b1f0-7898af800db1" path="/var/lib/kubelet/pods/7063ee78-5b1b-4f4a-b1f0-7898af800db1/volumes"
Oct 30 05:37:06.984041 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:06.984009    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:08.603739 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:08.603691    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:08.714252 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:08.714222    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:08.821750 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:08.821720    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:08.933832 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:08.933740    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9"]
Oct 30 05:37:08.934070 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:08.934021    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9" podUID="bfe8c234-c443-44a6-8420-9566768d6d02" containerName="istio-proxy" containerID="cri-o://2e2c53a2338a2d2dbc3f8266d0fce15a8e52a00618606f23f455b15c420cc0cd" gracePeriod=30
Oct 30 05:37:08.954258 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:08.954227    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:09.647213 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:09.647177    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:09.767721 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:09.767683    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:10.027128 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:10.027041    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb"]
Oct 30 05:37:10.027356 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:10.027298    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb" podUID="d77a7956-ec67-4cfb-ba98-fd1e6ea6f227" containerName="istio-proxy" containerID="cri-o://109ca16652c0ca0ffb819d91c285f062ab1096be49790b362fc1ce97b1ebd68c" gracePeriod=30
Oct 30 05:37:10.057310 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:10.057274    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:10.888911 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:10.888865    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:12.545407 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:12.545314    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:12.799975 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:12.799868    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:12.915479 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:12.915447    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:13.454725 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:13.454689    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:13.559340 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:13.559301    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:13.721178 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:13.721069    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:14.068115 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:14.068078    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:14.215539 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:14.215507    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9"
Oct 30 05:37:14.287295 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:14.287260    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfe8c234-c443-44a6-8420-9566768d6d02-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfe8c234-c443-44a6-8420-9566768d6d02-istio-envoy\") pod \"bfe8c234-c443-44a6-8420-9566768d6d02\" (UID: \"bfe8c234-c443-44a6-8420-9566768d6d02\") "
Oct 30 05:37:14.287546 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:14.287353    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfe8c234-c443-44a6-8420-9566768d6d02-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfe8c234-c443-44a6-8420-9566768d6d02-istio-data\") pod \"bfe8c234-c443-44a6-8420-9566768d6d02\" (UID: \"bfe8c234-c443-44a6-8420-9566768d6d02\") "
Oct 30 05:37:14.287546 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:14.287479    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfe8c234-c443-44a6-8420-9566768d6d02-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfe8c234-c443-44a6-8420-9566768d6d02-workload-certs\") pod \"bfe8c234-c443-44a6-8420-9566768d6d02\" (UID: \"bfe8c234-c443-44a6-8420-9566768d6d02\") "
Oct 30 05:37:14.287546 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:14.287505    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfe8c234-c443-44a6-8420-9566768d6d02-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfe8c234-c443-44a6-8420-9566768d6d02-istiod-ca-cert\") pod \"bfe8c234-c443-44a6-8420-9566768d6d02\" (UID: \"bfe8c234-c443-44a6-8420-9566768d6d02\") "
Oct 30 05:37:14.287751 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:14.287564    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfe8c234-c443-44a6-8420-9566768d6d02-kube-api-access-pk8kc\" (UniqueName: \"kubernetes.io/projected/bfe8c234-c443-44a6-8420-9566768d6d02-kube-api-access-pk8kc\") pod \"bfe8c234-c443-44a6-8420-9566768d6d02\" (UID: \"bfe8c234-c443-44a6-8420-9566768d6d02\") "
Oct 30 05:37:14.287751 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:14.287607    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bfe8c234-c443-44a6-8420-9566768d6d02-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfe8c234-c443-44a6-8420-9566768d6d02-istio-podinfo\") pod \"bfe8c234-c443-44a6-8420-9566768d6d02\" (UID: \"bfe8c234-c443-44a6-8420-9566768d6d02\") "
Oct 30 05:37:14.287751 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:14.287682    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfe8c234-c443-44a6-8420-9566768d6d02-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe8c234-c443-44a6-8420-9566768d6d02-credential-socket\") pod \"bfe8c234-c443-44a6-8420-9566768d6d02\" (UID: \"bfe8c234-c443-44a6-8420-9566768d6d02\") "
Oct 30 05:37:14.287751 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:14.287707    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfe8c234-c443-44a6-8420-9566768d6d02-istio-token\" (UniqueName: \"kubernetes.io/projected/bfe8c234-c443-44a6-8420-9566768d6d02-istio-token\") pod \"bfe8c234-c443-44a6-8420-9566768d6d02\" (UID: \"bfe8c234-c443-44a6-8420-9566768d6d02\") "
Oct 30 05:37:14.287751 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:14.287703    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfe8c234-c443-44a6-8420-9566768d6d02-istio-data" pod "bfe8c234-c443-44a6-8420-9566768d6d02" (UID: "bfe8c234-c443-44a6-8420-9566768d6d02"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:37:14.287751 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:14.287734    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfe8c234-c443-44a6-8420-9566768d6d02-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe8c234-c443-44a6-8420-9566768d6d02-workload-socket\") pod \"bfe8c234-c443-44a6-8420-9566768d6d02\" (UID: \"bfe8c234-c443-44a6-8420-9566768d6d02\") "
Oct 30 05:37:14.288141 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:14.288030    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfe8c234-c443-44a6-8420-9566768d6d02-credential-socket" pod "bfe8c234-c443-44a6-8420-9566768d6d02" (UID: "bfe8c234-c443-44a6-8420-9566768d6d02"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:37:14.288141 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:14.288090    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfe8c234-c443-44a6-8420-9566768d6d02-workload-certs" pod "bfe8c234-c443-44a6-8420-9566768d6d02" (UID: "bfe8c234-c443-44a6-8420-9566768d6d02"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:37:14.288253 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:14.288235    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfe8c234-c443-44a6-8420-9566768d6d02-workload-socket" pod "bfe8c234-c443-44a6-8420-9566768d6d02" (UID: "bfe8c234-c443-44a6-8420-9566768d6d02"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:37:14.288310 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:14.288283    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfe8c234-c443-44a6-8420-9566768d6d02-istiod-ca-cert" pod "bfe8c234-c443-44a6-8420-9566768d6d02" (UID: "bfe8c234-c443-44a6-8420-9566768d6d02"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:37:14.288454 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:14.288434    5898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe8c234-c443-44a6-8420-9566768d6d02-workload-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:37:14.288536 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:14.288459    5898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfe8c234-c443-44a6-8420-9566768d6d02-istio-data\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:37:14.288536 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:14.288474    5898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfe8c234-c443-44a6-8420-9566768d6d02-workload-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:37:14.288536 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:14.288487    5898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfe8c234-c443-44a6-8420-9566768d6d02-istiod-ca-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:37:14.288536 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:14.288500    5898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe8c234-c443-44a6-8420-9566768d6d02-credential-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:37:14.290315 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:14.290294    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfe8c234-c443-44a6-8420-9566768d6d02-istio-envoy" pod "bfe8c234-c443-44a6-8420-9566768d6d02" (UID: "bfe8c234-c443-44a6-8420-9566768d6d02"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:37:14.290418 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:14.290372    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bfe8c234-c443-44a6-8420-9566768d6d02-istio-podinfo" pod "bfe8c234-c443-44a6-8420-9566768d6d02" (UID: "bfe8c234-c443-44a6-8420-9566768d6d02"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:37:14.290772 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:14.290748    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfe8c234-c443-44a6-8420-9566768d6d02-istio-token" pod "bfe8c234-c443-44a6-8420-9566768d6d02" (UID: "bfe8c234-c443-44a6-8420-9566768d6d02"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:37:14.291009 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:14.290979    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfe8c234-c443-44a6-8420-9566768d6d02-kube-api-access-pk8kc" pod "bfe8c234-c443-44a6-8420-9566768d6d02" (UID: "bfe8c234-c443-44a6-8420-9566768d6d02"). InnerVolumeSpecName "kube-api-access-pk8kc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:37:14.322324 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:14.322291    5898 generic.go:356] "Generic (PLEG): container finished" podID="bfe8c234-c443-44a6-8420-9566768d6d02" containerID="2e2c53a2338a2d2dbc3f8266d0fce15a8e52a00618606f23f455b15c420cc0cd" exitCode=0
Oct 30 05:37:14.322464 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:14.322345    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9"
Oct 30 05:37:14.322464 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:14.322379    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9" event={"ID":"bfe8c234-c443-44a6-8420-9566768d6d02","Type":"ContainerDied","Data":"2e2c53a2338a2d2dbc3f8266d0fce15a8e52a00618606f23f455b15c420cc0cd"}
Oct 30 05:37:14.322577 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:14.322474    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9" event={"ID":"bfe8c234-c443-44a6-8420-9566768d6d02","Type":"ContainerDied","Data":"0610b4c9d3eb5f575067aed346d43cf749205e1dbe2edfb78fbbfe3acbb72f06"}
Oct 30 05:37:14.322577 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:14.322502    5898 scope.go:122] "RemoveContainer" containerID="2e2c53a2338a2d2dbc3f8266d0fce15a8e52a00618606f23f455b15c420cc0cd"
Oct 30 05:37:14.333304 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:14.333285    5898 scope.go:122] "RemoveContainer" containerID="2e2c53a2338a2d2dbc3f8266d0fce15a8e52a00618606f23f455b15c420cc0cd"
Oct 30 05:37:14.333593 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:37:14.333573    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e2c53a2338a2d2dbc3f8266d0fce15a8e52a00618606f23f455b15c420cc0cd\": container with ID starting with 2e2c53a2338a2d2dbc3f8266d0fce15a8e52a00618606f23f455b15c420cc0cd not found: ID does not exist" containerID="2e2c53a2338a2d2dbc3f8266d0fce15a8e52a00618606f23f455b15c420cc0cd"
Oct 30 05:37:14.333668 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:14.333603    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e2c53a2338a2d2dbc3f8266d0fce15a8e52a00618606f23f455b15c420cc0cd"} err="failed to get container status \"2e2c53a2338a2d2dbc3f8266d0fce15a8e52a00618606f23f455b15c420cc0cd\": rpc error: code = NotFound desc = could not find container \"2e2c53a2338a2d2dbc3f8266d0fce15a8e52a00618606f23f455b15c420cc0cd\": container with ID starting with 2e2c53a2338a2d2dbc3f8266d0fce15a8e52a00618606f23f455b15c420cc0cd not found: ID does not exist"
Oct 30 05:37:14.371996 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:14.371961    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9"]
Oct 30 05:37:14.380093 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:14.380066    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d55037-openshift-default-77bc79dc85-vn9p9"]
Oct 30 05:37:14.390369 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:14.390346    5898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfe8c234-c443-44a6-8420-9566768d6d02-istio-envoy\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:37:14.390369 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:14.390370    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pk8kc\" (UniqueName: \"kubernetes.io/projected/bfe8c234-c443-44a6-8420-9566768d6d02-kube-api-access-pk8kc\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:37:14.390520 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:14.390382    5898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfe8c234-c443-44a6-8420-9566768d6d02-istio-podinfo\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:37:14.390520 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:14.390392    5898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfe8c234-c443-44a6-8420-9566768d6d02-istio-token\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:37:14.890781 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:14.890748    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfe8c234-c443-44a6-8420-9566768d6d02" path="/var/lib/kubelet/pods/bfe8c234-c443-44a6-8420-9566768d6d02/volumes"
Oct 30 05:37:15.295312 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:15.295284    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb"
Oct 30 05:37:15.330806 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:15.330776    5898 generic.go:356] "Generic (PLEG): container finished" podID="d77a7956-ec67-4cfb-ba98-fd1e6ea6f227" containerID="109ca16652c0ca0ffb819d91c285f062ab1096be49790b362fc1ce97b1ebd68c" exitCode=0
Oct 30 05:37:15.330806 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:15.330810    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb" event={"ID":"d77a7956-ec67-4cfb-ba98-fd1e6ea6f227","Type":"ContainerDied","Data":"109ca16652c0ca0ffb819d91c285f062ab1096be49790b362fc1ce97b1ebd68c"}
Oct 30 05:37:15.331052 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:15.330821    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb"
Oct 30 05:37:15.331052 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:15.330853    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb" event={"ID":"d77a7956-ec67-4cfb-ba98-fd1e6ea6f227","Type":"ContainerDied","Data":"c5ea5f3d9bab3342ec250942960113cdb40f275995c90db3c7bd6591456c3af9"}
Oct 30 05:37:15.331052 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:15.330875    5898 scope.go:122] "RemoveContainer" containerID="109ca16652c0ca0ffb819d91c285f062ab1096be49790b362fc1ce97b1ebd68c"
Oct 30 05:37:15.340933 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:15.340917    5898 scope.go:122] "RemoveContainer" containerID="109ca16652c0ca0ffb819d91c285f062ab1096be49790b362fc1ce97b1ebd68c"
Oct 30 05:37:15.341199 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:37:15.341174    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"109ca16652c0ca0ffb819d91c285f062ab1096be49790b362fc1ce97b1ebd68c\": container with ID starting with 109ca16652c0ca0ffb819d91c285f062ab1096be49790b362fc1ce97b1ebd68c not found: ID does not exist" containerID="109ca16652c0ca0ffb819d91c285f062ab1096be49790b362fc1ce97b1ebd68c"
Oct 30 05:37:15.341262 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:15.341206    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"109ca16652c0ca0ffb819d91c285f062ab1096be49790b362fc1ce97b1ebd68c"} err="failed to get container status \"109ca16652c0ca0ffb819d91c285f062ab1096be49790b362fc1ce97b1ebd68c\": rpc error: code = NotFound desc = could not find container \"109ca16652c0ca0ffb819d91c285f062ab1096be49790b362fc1ce97b1ebd68c\": container with ID starting with 109ca16652c0ca0ffb819d91c285f062ab1096be49790b362fc1ce97b1ebd68c not found: ID does not exist"
Oct 30 05:37:15.404478 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:15.404401    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d77a7956-ec67-4cfb-ba98-fd1e6ea6f227-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d77a7956-ec67-4cfb-ba98-fd1e6ea6f227-istio-data\") pod \"d77a7956-ec67-4cfb-ba98-fd1e6ea6f227\" (UID: \"d77a7956-ec67-4cfb-ba98-fd1e6ea6f227\") "
Oct 30 05:37:15.404478 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:15.404458    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d77a7956-ec67-4cfb-ba98-fd1e6ea6f227-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d77a7956-ec67-4cfb-ba98-fd1e6ea6f227-istio-podinfo\") pod \"d77a7956-ec67-4cfb-ba98-fd1e6ea6f227\" (UID: \"d77a7956-ec67-4cfb-ba98-fd1e6ea6f227\") "
Oct 30 05:37:15.404719 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:15.404549    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d77a7956-ec67-4cfb-ba98-fd1e6ea6f227-kube-api-access-8tgp7\" (UniqueName: \"kubernetes.io/projected/d77a7956-ec67-4cfb-ba98-fd1e6ea6f227-kube-api-access-8tgp7\") pod \"d77a7956-ec67-4cfb-ba98-fd1e6ea6f227\" (UID: \"d77a7956-ec67-4cfb-ba98-fd1e6ea6f227\") "
Oct 30 05:37:15.404719 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:15.404591    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d77a7956-ec67-4cfb-ba98-fd1e6ea6f227-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d77a7956-ec67-4cfb-ba98-fd1e6ea6f227-credential-socket\") pod \"d77a7956-ec67-4cfb-ba98-fd1e6ea6f227\" (UID: \"d77a7956-ec67-4cfb-ba98-fd1e6ea6f227\") "
Oct 30 05:37:15.404719 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:15.404650    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d77a7956-ec67-4cfb-ba98-fd1e6ea6f227-istio-token\" (UniqueName: \"kubernetes.io/projected/d77a7956-ec67-4cfb-ba98-fd1e6ea6f227-istio-token\") pod \"d77a7956-ec67-4cfb-ba98-fd1e6ea6f227\" (UID: \"d77a7956-ec67-4cfb-ba98-fd1e6ea6f227\") "
Oct 30 05:37:15.404719 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:15.404707    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d77a7956-ec67-4cfb-ba98-fd1e6ea6f227-istio-data" pod "d77a7956-ec67-4cfb-ba98-fd1e6ea6f227" (UID: "d77a7956-ec67-4cfb-ba98-fd1e6ea6f227"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:37:15.405005 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:15.404735    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d77a7956-ec67-4cfb-ba98-fd1e6ea6f227-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d77a7956-ec67-4cfb-ba98-fd1e6ea6f227-workload-socket\") pod \"d77a7956-ec67-4cfb-ba98-fd1e6ea6f227\" (UID: \"d77a7956-ec67-4cfb-ba98-fd1e6ea6f227\") "
Oct 30 05:37:15.405005 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:15.404808    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d77a7956-ec67-4cfb-ba98-fd1e6ea6f227-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d77a7956-ec67-4cfb-ba98-fd1e6ea6f227-istiod-ca-cert\") pod \"d77a7956-ec67-4cfb-ba98-fd1e6ea6f227\" (UID: \"d77a7956-ec67-4cfb-ba98-fd1e6ea6f227\") "
Oct 30 05:37:15.405005 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:15.404838    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d77a7956-ec67-4cfb-ba98-fd1e6ea6f227-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d77a7956-ec67-4cfb-ba98-fd1e6ea6f227-workload-certs\") pod \"d77a7956-ec67-4cfb-ba98-fd1e6ea6f227\" (UID: \"d77a7956-ec67-4cfb-ba98-fd1e6ea6f227\") "
Oct 30 05:37:15.405005 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:15.404942    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d77a7956-ec67-4cfb-ba98-fd1e6ea6f227-credential-socket" pod "d77a7956-ec67-4cfb-ba98-fd1e6ea6f227" (UID: "d77a7956-ec67-4cfb-ba98-fd1e6ea6f227"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:37:15.405227 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:15.405024    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d77a7956-ec67-4cfb-ba98-fd1e6ea6f227-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d77a7956-ec67-4cfb-ba98-fd1e6ea6f227-istio-envoy\") pod \"d77a7956-ec67-4cfb-ba98-fd1e6ea6f227\" (UID: \"d77a7956-ec67-4cfb-ba98-fd1e6ea6f227\") "
Oct 30 05:37:15.405227 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:15.405184    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d77a7956-ec67-4cfb-ba98-fd1e6ea6f227-workload-certs" pod "d77a7956-ec67-4cfb-ba98-fd1e6ea6f227" (UID: "d77a7956-ec67-4cfb-ba98-fd1e6ea6f227"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:37:15.405352 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:15.405236    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d77a7956-ec67-4cfb-ba98-fd1e6ea6f227-workload-socket" pod "d77a7956-ec67-4cfb-ba98-fd1e6ea6f227" (UID: "d77a7956-ec67-4cfb-ba98-fd1e6ea6f227"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:37:15.405352 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:15.405300    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d77a7956-ec67-4cfb-ba98-fd1e6ea6f227-istiod-ca-cert" pod "d77a7956-ec67-4cfb-ba98-fd1e6ea6f227" (UID: "d77a7956-ec67-4cfb-ba98-fd1e6ea6f227"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:37:15.406009 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:15.405984    5898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d77a7956-ec67-4cfb-ba98-fd1e6ea6f227-istio-data\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:37:15.406106 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:15.406013    5898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d77a7956-ec67-4cfb-ba98-fd1e6ea6f227-credential-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:37:15.406106 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:15.406029    5898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d77a7956-ec67-4cfb-ba98-fd1e6ea6f227-workload-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:37:15.406106 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:15.406042    5898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d77a7956-ec67-4cfb-ba98-fd1e6ea6f227-istiod-ca-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:37:15.406106 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:15.406056    5898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d77a7956-ec67-4cfb-ba98-fd1e6ea6f227-workload-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:37:15.407176 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:15.407152    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d77a7956-ec67-4cfb-ba98-fd1e6ea6f227-istio-token" pod "d77a7956-ec67-4cfb-ba98-fd1e6ea6f227" (UID: "d77a7956-ec67-4cfb-ba98-fd1e6ea6f227"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:37:15.407343 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:15.407309    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d77a7956-ec67-4cfb-ba98-fd1e6ea6f227-istio-envoy" pod "d77a7956-ec67-4cfb-ba98-fd1e6ea6f227" (UID: "d77a7956-ec67-4cfb-ba98-fd1e6ea6f227"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:37:15.407434 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:15.407368    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d77a7956-ec67-4cfb-ba98-fd1e6ea6f227-kube-api-access-8tgp7" pod "d77a7956-ec67-4cfb-ba98-fd1e6ea6f227" (UID: "d77a7956-ec67-4cfb-ba98-fd1e6ea6f227"). InnerVolumeSpecName "kube-api-access-8tgp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:37:15.407434 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:15.407376    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d77a7956-ec67-4cfb-ba98-fd1e6ea6f227-istio-podinfo" pod "d77a7956-ec67-4cfb-ba98-fd1e6ea6f227" (UID: "d77a7956-ec67-4cfb-ba98-fd1e6ea6f227"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:37:15.507781 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:15.507744    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tgp7\" (UniqueName: \"kubernetes.io/projected/d77a7956-ec67-4cfb-ba98-fd1e6ea6f227-kube-api-access-8tgp7\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:37:15.507781 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:15.507775    5898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d77a7956-ec67-4cfb-ba98-fd1e6ea6f227-istio-token\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:37:15.507781 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:15.507785    5898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d77a7956-ec67-4cfb-ba98-fd1e6ea6f227-istio-envoy\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:37:15.507781 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:15.507793    5898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d77a7956-ec67-4cfb-ba98-fd1e6ea6f227-istio-podinfo\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:37:15.573551 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:15.573514    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:15.713673 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:15.713637    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb"]
Oct 30 05:37:15.719300 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:15.719268    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ff4760-openshift-default-7bb6ff757-qj5pb"]
Oct 30 05:37:16.891411 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:16.891380    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d77a7956-ec67-4cfb-ba98-fd1e6ea6f227" path="/var/lib/kubelet/pods/d77a7956-ec67-4cfb-ba98-fd1e6ea6f227/volumes"
Oct 30 05:37:17.751114 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:17.751079    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:19.142190 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:19.142146    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:19.346443 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:19.346408    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:20.403137 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:20.403102    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:20.550785 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:20.550750    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:20.807717 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:20.807636    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:21.348143 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:21.348109    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:21.462922 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:21.462815    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:23.070926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:23.070337    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:37:23.279659 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:23.279624    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:25.111927 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:25.111887    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:26.818892 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:37:26.818855    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:37:26.961703 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:26.961664    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:28.815842 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:28.815807    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:30.778755 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:30.778694    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:31.789378 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:31.789343    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-44bc69-5784cbb69b-68q6q"]
Oct 30 05:37:31.825251 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:31.825216    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-44bc69-59d6bdddbc-c58st"]
Oct 30 05:37:31.951675 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:31.951646    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:32.086790 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:32.086751    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:32.698582 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:32.698532    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:34.457985 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:34.457951    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:36.529576 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:36.529538    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:36.790595 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:36.790489    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-44bc69-5784cbb69b-68q6q" podUID="3270ef0e-4775-48c6-8724-baff8e3fac06" containerName="payload-processing" containerID="cri-o://c84467b0e29b66363e3614465fd70c9845a14d26565a777c18932e81d947f6a6" gracePeriod=25
Oct 30 05:37:36.826388 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:36.826351    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-44bc69-59d6bdddbc-c58st" podUID="fc35e0f4-fbc7-43e8-bf45-10d4d3ff19fe" containerName="payload-pre-processing" containerID="cri-o://c3469c41e58626e4c573f5c2be26f18bcb97261e12f136cb68c7002bc6f24893" gracePeriod=25
Oct 30 05:37:38.899848 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:38.899814    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:41.540400 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:41.540364    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:42.890992 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:42.890950    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-44bc69-5784cbb69b-68q6q" podUID="3270ef0e-4775-48c6-8724-baff8e3fac06" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:37:42.890992 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:42.890974    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-44bc69-59d6bdddbc-c58st" podUID="fc35e0f4-fbc7-43e8-bf45-10d4d3ff19fe" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:37:43.398713 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:43.398683    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:43.746417 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:43.746326    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:43.891102 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:43.891065    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:45.461617 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:45.461587    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:47.183318 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:47.183280    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:47.567074 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:47.566985    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dfb231-8574fd944b-hd2hq"]
Oct 30 05:37:47.567565 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:47.567507    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dfb231-8574fd944b-hd2hq" podUID="b35f68e0-c492-44ba-a145-026a63713b60" containerName="maas-api" containerID="cri-o://b9f2c05280b349e089bbe5d1658649c0c78a48f7e380fc6e1bad98b6404ab663" gracePeriod=30
Oct 30 05:37:47.647589 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:47.647552    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:47.809934 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:47.805093    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:47.975757 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:47.975730    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dfb231-8574fd944b-hd2hq"
Oct 30 05:37:48.019924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:48.009705    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:48.085059 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:48.085025    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b35f68e0-c492-44ba-a145-026a63713b60-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b35f68e0-c492-44ba-a145-026a63713b60-openshift-service-ca\") pod \"b35f68e0-c492-44ba-a145-026a63713b60\" (UID: \"b35f68e0-c492-44ba-a145-026a63713b60\") "
Oct 30 05:37:48.085339 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:48.085318    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b35f68e0-c492-44ba-a145-026a63713b60-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b35f68e0-c492-44ba-a145-026a63713b60-metrics-certs\") pod \"b35f68e0-c492-44ba-a145-026a63713b60\" (UID: \"b35f68e0-c492-44ba-a145-026a63713b60\") "
Oct 30 05:37:48.085490 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:48.085473    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b35f68e0-c492-44ba-a145-026a63713b60-kube-api-access-rpqh5\" (UniqueName: \"kubernetes.io/projected/b35f68e0-c492-44ba-a145-026a63713b60-kube-api-access-rpqh5\") pod \"b35f68e0-c492-44ba-a145-026a63713b60\" (UID: \"b35f68e0-c492-44ba-a145-026a63713b60\") "
Oct 30 05:37:48.085775 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:48.085758    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b35f68e0-c492-44ba-a145-026a63713b60-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b35f68e0-c492-44ba-a145-026a63713b60-maas-api-tls\") pod \"b35f68e0-c492-44ba-a145-026a63713b60\" (UID: \"b35f68e0-c492-44ba-a145-026a63713b60\") "
Oct 30 05:37:48.094193 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:48.093310    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b35f68e0-c492-44ba-a145-026a63713b60-openshift-service-ca" pod "b35f68e0-c492-44ba-a145-026a63713b60" (UID: "b35f68e0-c492-44ba-a145-026a63713b60"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:37:48.098784 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:48.098312    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b35f68e0-c492-44ba-a145-026a63713b60-kube-api-access-rpqh5" pod "b35f68e0-c492-44ba-a145-026a63713b60" (UID: "b35f68e0-c492-44ba-a145-026a63713b60"). InnerVolumeSpecName "kube-api-access-rpqh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:37:48.113243 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:48.107016    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b35f68e0-c492-44ba-a145-026a63713b60-maas-api-tls" pod "b35f68e0-c492-44ba-a145-026a63713b60" (UID: "b35f68e0-c492-44ba-a145-026a63713b60"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:37:48.113243 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:48.107093    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b35f68e0-c492-44ba-a145-026a63713b60-metrics-certs" pod "b35f68e0-c492-44ba-a145-026a63713b60" (UID: "b35f68e0-c492-44ba-a145-026a63713b60"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:37:48.188196 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:48.188086    5898 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b35f68e0-c492-44ba-a145-026a63713b60-openshift-service-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:37:48.188196 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:48.188127    5898 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b35f68e0-c492-44ba-a145-026a63713b60-metrics-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:37:48.188196 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:48.188141    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpqh5\" (UniqueName: \"kubernetes.io/projected/b35f68e0-c492-44ba-a145-026a63713b60-kube-api-access-rpqh5\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:37:48.188196 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:48.188157    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b35f68e0-c492-44ba-a145-026a63713b60-maas-api-tls\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:37:48.452743 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:48.452664    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:48.599153 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:48.599119    5898 generic.go:356] "Generic (PLEG): container finished" podID="b35f68e0-c492-44ba-a145-026a63713b60" containerID="b9f2c05280b349e089bbe5d1658649c0c78a48f7e380fc6e1bad98b6404ab663" exitCode=0
Oct 30 05:37:48.599351 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:48.599177    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dfb231-8574fd944b-hd2hq"
Oct 30 05:37:48.599351 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:48.599177    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dfb231-8574fd944b-hd2hq" event={"ID":"b35f68e0-c492-44ba-a145-026a63713b60","Type":"ContainerDied","Data":"b9f2c05280b349e089bbe5d1658649c0c78a48f7e380fc6e1bad98b6404ab663"}
Oct 30 05:37:48.599351 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:48.599284    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dfb231-8574fd944b-hd2hq" event={"ID":"b35f68e0-c492-44ba-a145-026a63713b60","Type":"ContainerDied","Data":"fd7620073d2ddd17093c5c4d3a90fb5dd538e04bccb11074a3b9a27437665beb"}
Oct 30 05:37:48.599351 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:48.599318    5898 scope.go:122] "RemoveContainer" containerID="b9f2c05280b349e089bbe5d1658649c0c78a48f7e380fc6e1bad98b6404ab663"
Oct 30 05:37:48.599596 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:48.599531    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:48.637143 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:48.637120    5898 scope.go:122] "RemoveContainer" containerID="b9f2c05280b349e089bbe5d1658649c0c78a48f7e380fc6e1bad98b6404ab663"
Oct 30 05:37:48.637465 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:37:48.637442    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9f2c05280b349e089bbe5d1658649c0c78a48f7e380fc6e1bad98b6404ab663\": container with ID starting with b9f2c05280b349e089bbe5d1658649c0c78a48f7e380fc6e1bad98b6404ab663 not found: ID does not exist" containerID="b9f2c05280b349e089bbe5d1658649c0c78a48f7e380fc6e1bad98b6404ab663"
Oct 30 05:37:48.637518 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:48.637479    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9f2c05280b349e089bbe5d1658649c0c78a48f7e380fc6e1bad98b6404ab663"} err="failed to get container status \"b9f2c05280b349e089bbe5d1658649c0c78a48f7e380fc6e1bad98b6404ab663\": rpc error: code = NotFound desc = could not find container \"b9f2c05280b349e089bbe5d1658649c0c78a48f7e380fc6e1bad98b6404ab663\": container with ID starting with b9f2c05280b349e089bbe5d1658649c0c78a48f7e380fc6e1bad98b6404ab663 not found: ID does not exist"
Oct 30 05:37:48.672980 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:48.672952    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dfb231-8574fd944b-hd2hq"]
Oct 30 05:37:48.683344 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:48.683321    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dfb231-8574fd944b-hd2hq"]
Oct 30 05:37:48.754315 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:48.754090    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:48.890517 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:48.890477    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b35f68e0-c492-44ba-a145-026a63713b60" path="/var/lib/kubelet/pods/b35f68e0-c492-44ba-a145-026a63713b60/volumes"
Oct 30 05:37:49.163798 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:49.163748    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:51.241395 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:51.241356    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:52.891197 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:52.891164    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-44bc69-59d6bdddbc-c58st" podUID="fc35e0f4-fbc7-43e8-bf45-10d4d3ff19fe" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:37:52.891602 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:52.891167    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-44bc69-5784cbb69b-68q6q" podUID="3270ef0e-4775-48c6-8724-baff8e3fac06" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:37:53.734803 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:53.734760    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:55.616517 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:55.616480    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:55.981109 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:55.981035    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:56.092393 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:56.092357    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:56.352547 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:56.352516    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:56.818717 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:37:56.818686    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:37:57.040857 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:57.040820    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:37:57.050392 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:57.050280    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:57.220506 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:57.220469    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:57.233468 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:57.233429    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-44bc69-5784cbb69b-68q6q"
Oct 30 05:37:57.241113 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:57.241086    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-44bc69-59d6bdddbc-c58st"
Oct 30 05:37:57.345735 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:57.344860    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3270ef0e-4775-48c6-8724-baff8e3fac06-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3270ef0e-4775-48c6-8724-baff8e3fac06-plugins-config-volume\") pod \"3270ef0e-4775-48c6-8724-baff8e3fac06\" (UID: \"3270ef0e-4775-48c6-8724-baff8e3fac06\") "
Oct 30 05:37:57.345735 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:57.344941    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc35e0f4-fbc7-43e8-bf45-10d4d3ff19fe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc35e0f4-fbc7-43e8-bf45-10d4d3ff19fe-plugins-config-volume\") pod \"fc35e0f4-fbc7-43e8-bf45-10d4d3ff19fe\" (UID: \"fc35e0f4-fbc7-43e8-bf45-10d4d3ff19fe\") "
Oct 30 05:37:57.345735 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:57.345034    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc35e0f4-fbc7-43e8-bf45-10d4d3ff19fe-kube-api-access-sfp9t\" (UniqueName: \"kubernetes.io/projected/fc35e0f4-fbc7-43e8-bf45-10d4d3ff19fe-kube-api-access-sfp9t\") pod \"fc35e0f4-fbc7-43e8-bf45-10d4d3ff19fe\" (UID: \"fc35e0f4-fbc7-43e8-bf45-10d4d3ff19fe\") "
Oct 30 05:37:57.345735 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:57.345061    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3270ef0e-4775-48c6-8724-baff8e3fac06-kube-api-access-fq2gf\" (UniqueName: \"kubernetes.io/projected/3270ef0e-4775-48c6-8724-baff8e3fac06-kube-api-access-fq2gf\") pod \"3270ef0e-4775-48c6-8724-baff8e3fac06\" (UID: \"3270ef0e-4775-48c6-8724-baff8e3fac06\") "
Oct 30 05:37:57.346419 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:57.346393    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3270ef0e-4775-48c6-8724-baff8e3fac06-plugins-config-volume" pod "3270ef0e-4775-48c6-8724-baff8e3fac06" (UID: "3270ef0e-4775-48c6-8724-baff8e3fac06"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:37:57.347223 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:57.347198    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc35e0f4-fbc7-43e8-bf45-10d4d3ff19fe-plugins-config-volume" pod "fc35e0f4-fbc7-43e8-bf45-10d4d3ff19fe" (UID: "fc35e0f4-fbc7-43e8-bf45-10d4d3ff19fe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:37:57.347481 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:57.347461    5898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3270ef0e-4775-48c6-8724-baff8e3fac06-plugins-config-volume\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:37:57.347561 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:57.347486    5898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc35e0f4-fbc7-43e8-bf45-10d4d3ff19fe-plugins-config-volume\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:37:57.349758 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:57.349730    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3270ef0e-4775-48c6-8724-baff8e3fac06-kube-api-access-fq2gf" pod "3270ef0e-4775-48c6-8724-baff8e3fac06" (UID: "3270ef0e-4775-48c6-8724-baff8e3fac06"). InnerVolumeSpecName "kube-api-access-fq2gf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:37:57.352216 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:57.352189    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc35e0f4-fbc7-43e8-bf45-10d4d3ff19fe-kube-api-access-sfp9t" pod "fc35e0f4-fbc7-43e8-bf45-10d4d3ff19fe" (UID: "fc35e0f4-fbc7-43e8-bf45-10d4d3ff19fe"). InnerVolumeSpecName "kube-api-access-sfp9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:37:57.363150 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:57.362834    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd"]
Oct 30 05:37:57.363150 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:57.363065    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd" podUID="d4fd7194-084b-4339-9ea3-c5601858fed5" containerName="istio-proxy" containerID="cri-o://19841e4f8accb3a6b8f541099d9cbafdc222df3c6d16369aed86662c69c57f58" gracePeriod=30
Oct 30 05:37:57.429510 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:57.429478    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:57.449174 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:57.449150    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfp9t\" (UniqueName: \"kubernetes.io/projected/fc35e0f4-fbc7-43e8-bf45-10d4d3ff19fe-kube-api-access-sfp9t\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:37:57.449174 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:57.449175    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq2gf\" (UniqueName: \"kubernetes.io/projected/3270ef0e-4775-48c6-8724-baff8e3fac06-kube-api-access-fq2gf\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:37:57.667451 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:57.667363    5898 generic.go:356] "Generic (PLEG): container finished" podID="3270ef0e-4775-48c6-8724-baff8e3fac06" containerID="c84467b0e29b66363e3614465fd70c9845a14d26565a777c18932e81d947f6a6" exitCode=0
Oct 30 05:37:57.667451 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:57.667419    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-44bc69-5784cbb69b-68q6q"
Oct 30 05:37:57.667716 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:57.667448    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-44bc69-5784cbb69b-68q6q" event={"ID":"3270ef0e-4775-48c6-8724-baff8e3fac06","Type":"ContainerDied","Data":"c84467b0e29b66363e3614465fd70c9845a14d26565a777c18932e81d947f6a6"}
Oct 30 05:37:57.667716 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:57.667525    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-44bc69-5784cbb69b-68q6q" event={"ID":"3270ef0e-4775-48c6-8724-baff8e3fac06","Type":"ContainerDied","Data":"1ed60c8b57a93ac85f7d85d5a074ae4b9e55bf41fe53514ceb8275d23a06c1cc"}
Oct 30 05:37:57.667716 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:57.667565    5898 scope.go:122] "RemoveContainer" containerID="c84467b0e29b66363e3614465fd70c9845a14d26565a777c18932e81d947f6a6"
Oct 30 05:37:57.668813 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:57.668789    5898 generic.go:356] "Generic (PLEG): container finished" podID="fc35e0f4-fbc7-43e8-bf45-10d4d3ff19fe" containerID="c3469c41e58626e4c573f5c2be26f18bcb97261e12f136cb68c7002bc6f24893" exitCode=0
Oct 30 05:37:57.668952 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:57.668842    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-44bc69-59d6bdddbc-c58st" event={"ID":"fc35e0f4-fbc7-43e8-bf45-10d4d3ff19fe","Type":"ContainerDied","Data":"c3469c41e58626e4c573f5c2be26f18bcb97261e12f136cb68c7002bc6f24893"}
Oct 30 05:37:57.668952 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:57.668859    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-44bc69-59d6bdddbc-c58st"
Oct 30 05:37:57.668952 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:57.668883    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-44bc69-59d6bdddbc-c58st" event={"ID":"fc35e0f4-fbc7-43e8-bf45-10d4d3ff19fe","Type":"ContainerDied","Data":"260e3379a1a3147210e722fde71dcdda33828a3651a56531c53c0a6e92d77b43"}
Oct 30 05:37:57.675737 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:57.675705    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:57.677791 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:57.677775    5898 scope.go:122] "RemoveContainer" containerID="c84467b0e29b66363e3614465fd70c9845a14d26565a777c18932e81d947f6a6"
Oct 30 05:37:57.678124 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:37:57.678105    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c84467b0e29b66363e3614465fd70c9845a14d26565a777c18932e81d947f6a6\": container with ID starting with c84467b0e29b66363e3614465fd70c9845a14d26565a777c18932e81d947f6a6 not found: ID does not exist" containerID="c84467b0e29b66363e3614465fd70c9845a14d26565a777c18932e81d947f6a6"
Oct 30 05:37:57.678200 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:57.678134    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c84467b0e29b66363e3614465fd70c9845a14d26565a777c18932e81d947f6a6"} err="failed to get container status \"c84467b0e29b66363e3614465fd70c9845a14d26565a777c18932e81d947f6a6\": rpc error: code = NotFound desc = could not find container \"c84467b0e29b66363e3614465fd70c9845a14d26565a777c18932e81d947f6a6\": container with ID starting with c84467b0e29b66363e3614465fd70c9845a14d26565a777c18932e81d947f6a6 not found: ID does not exist"
Oct 30 05:37:57.678200 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:57.678150    5898 scope.go:122] "RemoveContainer" containerID="c3469c41e58626e4c573f5c2be26f18bcb97261e12f136cb68c7002bc6f24893"
Oct 30 05:37:57.686896 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:57.686879    5898 scope.go:122] "RemoveContainer" containerID="c3469c41e58626e4c573f5c2be26f18bcb97261e12f136cb68c7002bc6f24893"
Oct 30 05:37:57.687189 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:37:57.687170    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3469c41e58626e4c573f5c2be26f18bcb97261e12f136cb68c7002bc6f24893\": container with ID starting with c3469c41e58626e4c573f5c2be26f18bcb97261e12f136cb68c7002bc6f24893 not found: ID does not exist" containerID="c3469c41e58626e4c573f5c2be26f18bcb97261e12f136cb68c7002bc6f24893"
Oct 30 05:37:57.687243 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:57.687198    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3469c41e58626e4c573f5c2be26f18bcb97261e12f136cb68c7002bc6f24893"} err="failed to get container status \"c3469c41e58626e4c573f5c2be26f18bcb97261e12f136cb68c7002bc6f24893\": rpc error: code = NotFound desc = could not find container \"c3469c41e58626e4c573f5c2be26f18bcb97261e12f136cb68c7002bc6f24893\": container with ID starting with c3469c41e58626e4c573f5c2be26f18bcb97261e12f136cb68c7002bc6f24893 not found: ID does not exist"
Oct 30 05:37:57.735962 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:57.729513    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-44bc69-5784cbb69b-68q6q"]
Oct 30 05:37:57.736358 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:57.736312    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-44bc69-5784cbb69b-68q6q"]
Oct 30 05:37:57.772725 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:57.772691    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-44bc69-59d6bdddbc-c58st"]
Oct 30 05:37:57.777937 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:57.777909    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-44bc69-59d6bdddbc-c58st"]
Oct 30 05:37:58.137279 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:58.137246    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:37:58.890765 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:58.890730    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3270ef0e-4775-48c6-8724-baff8e3fac06" path="/var/lib/kubelet/pods/3270ef0e-4775-48c6-8724-baff8e3fac06/volumes"
Oct 30 05:37:58.891178 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:58.891162    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc35e0f4-fbc7-43e8-bf45-10d4d3ff19fe" path="/var/lib/kubelet/pods/fc35e0f4-fbc7-43e8-bf45-10d4d3ff19fe/volumes"
Oct 30 05:37:59.981967 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:37:59.981936    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:01.786424 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:01.786396    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:02.637793 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:02.637737    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd"
Oct 30 05:38:02.708113 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:02.708082    5898 generic.go:356] "Generic (PLEG): container finished" podID="d4fd7194-084b-4339-9ea3-c5601858fed5" containerID="19841e4f8accb3a6b8f541099d9cbafdc222df3c6d16369aed86662c69c57f58" exitCode=0
Oct 30 05:38:02.708299 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:02.708127    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd"
Oct 30 05:38:02.708299 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:02.708167    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd" event={"ID":"d4fd7194-084b-4339-9ea3-c5601858fed5","Type":"ContainerDied","Data":"19841e4f8accb3a6b8f541099d9cbafdc222df3c6d16369aed86662c69c57f58"}
Oct 30 05:38:02.708299 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:02.708266    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd" event={"ID":"d4fd7194-084b-4339-9ea3-c5601858fed5","Type":"ContainerDied","Data":"60807dc4fea3e3a2a31b9068ee212a5cd67e18b16f49189a9941ae3b96fcfff8"}
Oct 30 05:38:02.708299 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:02.708297    5898 scope.go:122] "RemoveContainer" containerID="19841e4f8accb3a6b8f541099d9cbafdc222df3c6d16369aed86662c69c57f58"
Oct 30 05:38:02.715680 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:02.715659    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4fd7194-084b-4339-9ea3-c5601858fed5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4fd7194-084b-4339-9ea3-c5601858fed5-workload-certs\") pod \"d4fd7194-084b-4339-9ea3-c5601858fed5\" (UID: \"d4fd7194-084b-4339-9ea3-c5601858fed5\") "
Oct 30 05:38:02.715781 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:02.715716    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4fd7194-084b-4339-9ea3-c5601858fed5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4fd7194-084b-4339-9ea3-c5601858fed5-credential-socket\") pod \"d4fd7194-084b-4339-9ea3-c5601858fed5\" (UID: \"d4fd7194-084b-4339-9ea3-c5601858fed5\") "
Oct 30 05:38:02.715781 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:02.715735    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4fd7194-084b-4339-9ea3-c5601858fed5-istio-token\" (UniqueName: \"kubernetes.io/projected/d4fd7194-084b-4339-9ea3-c5601858fed5-istio-token\") pod \"d4fd7194-084b-4339-9ea3-c5601858fed5\" (UID: \"d4fd7194-084b-4339-9ea3-c5601858fed5\") "
Oct 30 05:38:02.715880 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:02.715821    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4fd7194-084b-4339-9ea3-c5601858fed5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4fd7194-084b-4339-9ea3-c5601858fed5-istiod-ca-cert\") pod \"d4fd7194-084b-4339-9ea3-c5601858fed5\" (UID: \"d4fd7194-084b-4339-9ea3-c5601858fed5\") "
Oct 30 05:38:02.715952 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:02.715885    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4fd7194-084b-4339-9ea3-c5601858fed5-kube-api-access-xgncq\" (UniqueName: \"kubernetes.io/projected/d4fd7194-084b-4339-9ea3-c5601858fed5-kube-api-access-xgncq\") pod \"d4fd7194-084b-4339-9ea3-c5601858fed5\" (UID: \"d4fd7194-084b-4339-9ea3-c5601858fed5\") "
Oct 30 05:38:02.715952 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:02.715938    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4fd7194-084b-4339-9ea3-c5601858fed5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4fd7194-084b-4339-9ea3-c5601858fed5-workload-socket\") pod \"d4fd7194-084b-4339-9ea3-c5601858fed5\" (UID: \"d4fd7194-084b-4339-9ea3-c5601858fed5\") "
Oct 30 05:38:02.716060 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:02.715964    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4fd7194-084b-4339-9ea3-c5601858fed5-workload-certs" pod "d4fd7194-084b-4339-9ea3-c5601858fed5" (UID: "d4fd7194-084b-4339-9ea3-c5601858fed5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:38:02.716060 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:02.715972    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4fd7194-084b-4339-9ea3-c5601858fed5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4fd7194-084b-4339-9ea3-c5601858fed5-istio-data\") pod \"d4fd7194-084b-4339-9ea3-c5601858fed5\" (UID: \"d4fd7194-084b-4339-9ea3-c5601858fed5\") "
Oct 30 05:38:02.716156 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:02.716074    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4fd7194-084b-4339-9ea3-c5601858fed5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4fd7194-084b-4339-9ea3-c5601858fed5-istio-envoy\") pod \"d4fd7194-084b-4339-9ea3-c5601858fed5\" (UID: \"d4fd7194-084b-4339-9ea3-c5601858fed5\") "
Oct 30 05:38:02.716156 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:02.716116    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4fd7194-084b-4339-9ea3-c5601858fed5-credential-socket" pod "d4fd7194-084b-4339-9ea3-c5601858fed5" (UID: "d4fd7194-084b-4339-9ea3-c5601858fed5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:38:02.716156 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:02.716126    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d4fd7194-084b-4339-9ea3-c5601858fed5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4fd7194-084b-4339-9ea3-c5601858fed5-istio-podinfo\") pod \"d4fd7194-084b-4339-9ea3-c5601858fed5\" (UID: \"d4fd7194-084b-4339-9ea3-c5601858fed5\") "
Oct 30 05:38:02.716331 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:02.716301    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4fd7194-084b-4339-9ea3-c5601858fed5-istio-data" pod "d4fd7194-084b-4339-9ea3-c5601858fed5" (UID: "d4fd7194-084b-4339-9ea3-c5601858fed5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:38:02.716331 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:02.716313    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4fd7194-084b-4339-9ea3-c5601858fed5-istiod-ca-cert" pod "d4fd7194-084b-4339-9ea3-c5601858fed5" (UID: "d4fd7194-084b-4339-9ea3-c5601858fed5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:38:02.716455 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:02.716314    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4fd7194-084b-4339-9ea3-c5601858fed5-workload-socket" pod "d4fd7194-084b-4339-9ea3-c5601858fed5" (UID: "d4fd7194-084b-4339-9ea3-c5601858fed5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:38:02.717184 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:02.717158    5898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4fd7194-084b-4339-9ea3-c5601858fed5-workload-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:38:02.717342 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:02.717189    5898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4fd7194-084b-4339-9ea3-c5601858fed5-credential-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:38:02.717342 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:02.717206    5898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4fd7194-084b-4339-9ea3-c5601858fed5-istiod-ca-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:38:02.717342 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:02.717222    5898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4fd7194-084b-4339-9ea3-c5601858fed5-workload-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:38:02.717342 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:02.717235    5898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4fd7194-084b-4339-9ea3-c5601858fed5-istio-data\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:38:02.718266 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:02.718244    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4fd7194-084b-4339-9ea3-c5601858fed5-istio-podinfo" pod "d4fd7194-084b-4339-9ea3-c5601858fed5" (UID: "d4fd7194-084b-4339-9ea3-c5601858fed5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:38:02.718441 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:02.718427    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4fd7194-084b-4339-9ea3-c5601858fed5-istio-token" pod "d4fd7194-084b-4339-9ea3-c5601858fed5" (UID: "d4fd7194-084b-4339-9ea3-c5601858fed5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:38:02.718587 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:02.718566    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4fd7194-084b-4339-9ea3-c5601858fed5-kube-api-access-xgncq" pod "d4fd7194-084b-4339-9ea3-c5601858fed5" (UID: "d4fd7194-084b-4339-9ea3-c5601858fed5"). InnerVolumeSpecName "kube-api-access-xgncq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:38:02.718648 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:02.718622    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4fd7194-084b-4339-9ea3-c5601858fed5-istio-envoy" pod "d4fd7194-084b-4339-9ea3-c5601858fed5" (UID: "d4fd7194-084b-4339-9ea3-c5601858fed5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:38:02.719128 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:02.719113    5898 scope.go:122] "RemoveContainer" containerID="19841e4f8accb3a6b8f541099d9cbafdc222df3c6d16369aed86662c69c57f58"
Oct 30 05:38:02.719412 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:38:02.719393    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19841e4f8accb3a6b8f541099d9cbafdc222df3c6d16369aed86662c69c57f58\": container with ID starting with 19841e4f8accb3a6b8f541099d9cbafdc222df3c6d16369aed86662c69c57f58 not found: ID does not exist" containerID="19841e4f8accb3a6b8f541099d9cbafdc222df3c6d16369aed86662c69c57f58"
Oct 30 05:38:02.719474 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:02.719419    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19841e4f8accb3a6b8f541099d9cbafdc222df3c6d16369aed86662c69c57f58"} err="failed to get container status \"19841e4f8accb3a6b8f541099d9cbafdc222df3c6d16369aed86662c69c57f58\": rpc error: code = NotFound desc = could not find container \"19841e4f8accb3a6b8f541099d9cbafdc222df3c6d16369aed86662c69c57f58\": container with ID starting with 19841e4f8accb3a6b8f541099d9cbafdc222df3c6d16369aed86662c69c57f58 not found: ID does not exist"
Oct 30 05:38:02.818461 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:02.818427    5898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4fd7194-084b-4339-9ea3-c5601858fed5-istio-token\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:38:02.818461 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:02.818457    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgncq\" (UniqueName: \"kubernetes.io/projected/d4fd7194-084b-4339-9ea3-c5601858fed5-kube-api-access-xgncq\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:38:02.818461 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:02.818466    5898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4fd7194-084b-4339-9ea3-c5601858fed5-istio-envoy\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:38:02.818987 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:02.818477    5898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4fd7194-084b-4339-9ea3-c5601858fed5-istio-podinfo\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:38:03.049507 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:03.049468    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd"]
Oct 30 05:38:03.060660 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:03.060635    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-dfb231-openshift-default-84687b4bd5-ftcbd"]
Oct 30 05:38:03.103507 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:03.103480    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6"]
Oct 30 05:38:03.116850 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:03.116819    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh"]
Oct 30 05:38:03.130048 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:03.130017    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx"]
Oct 30 05:38:03.264638 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:03.264554    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:03.913095 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:03.913065    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:04.890827 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:04.890794    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4fd7194-084b-4339-9ea3-c5601858fed5" path="/var/lib/kubelet/pods/d4fd7194-084b-4339-9ea3-c5601858fed5/volumes"
Oct 30 05:38:05.815556 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:05.815519    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:07.506518 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:07.506482    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:07.631874 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:07.631841    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:07.780985 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:07.767639    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:08.821389 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:08.821353    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b21507-59b47dc888-29s5k"]
Oct 30 05:38:08.850985 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:08.850950    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b21507-b7c9cc4c5-bkrt5"]
Oct 30 05:38:09.016962 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:09.016928    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:09.135343 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:09.135300    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:09.255814 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:09.255782    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:09.979807 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:09.979768    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:10.210745 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:10.210709    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:10.346424 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:10.346393    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:10.844100 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:10.844068    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:10.970540 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:10.970502    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:11.089721 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:11.089691    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:11.480813 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:11.480788    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:13.002921 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:13.002876    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:13.822069 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:13.822029    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-b21507-59b47dc888-29s5k" podUID="30460201-04af-49fb-a7d3-5ee121e3bdb4" containerName="payload-processing" containerID="cri-o://4ba9604e0c21f1b6d817fe4ca8f0099336f3de54d1b09c6a9ac0170423555310" gracePeriod=25
Oct 30 05:38:13.851687 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:13.851649    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b21507-b7c9cc4c5-bkrt5" podUID="c00385be-b173-422b-8ba6-0985c9cf7598" containerName="payload-pre-processing" containerID="cri-o://b37375a2fb7eaaf4020e964ad8c46fb6f2cee83c7c4ab28a874c230648b93568" gracePeriod=25
Oct 30 05:38:14.792026 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:14.791997    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:16.669196 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:16.669162    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:18.019625 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.019592    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:18.124598 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.124561    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6" podUID="b09804e3-8135-4f6d-ae34-c45d6da9728e" containerName="main" containerID="cri-o://28741bfcd2964321cba1ee5775c787f0ab4c32774a47338b54efd3b2d9c05fc4" gracePeriod=45
Oct 30 05:38:18.137273 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.137241    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh" podUID="38428a53-04c7-400a-9c78-52407e982478" containerName="main" containerID="cri-o://b702a116520d68edde52842852fcb463f2966912049c96ba49d8814ee70099e0" gracePeriod=45
Oct 30 05:38:18.140455 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.140423    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:18.149737 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.149707    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx" podUID="d523008f-c740-4a41-89cb-e82e1ffa0094" containerName="main" containerID="cri-o://09527f87c929c6f43806add434b18df957d709cc967f2aba2be1e376fc61e00f" gracePeriod=45
Oct 30 05:38:18.234142 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.234109    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx" podUID="d523008f-c740-4a41-89cb-e82e1ffa0094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": dial tcp 10.133.0.45:8000: connect: connection refused"
Oct 30 05:38:18.387105 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.387070    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:18.518969 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.513119    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh"
Oct 30 05:38:18.605922 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.600392    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38428a53-04c7-400a-9c78-52407e982478-home\" (UniqueName: \"kubernetes.io/empty-dir/38428a53-04c7-400a-9c78-52407e982478-home\") pod \"38428a53-04c7-400a-9c78-52407e982478\" (UID: \"38428a53-04c7-400a-9c78-52407e982478\") "
Oct 30 05:38:18.605922 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.600439    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38428a53-04c7-400a-9c78-52407e982478-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38428a53-04c7-400a-9c78-52407e982478-model-cache\") pod \"38428a53-04c7-400a-9c78-52407e982478\" (UID: \"38428a53-04c7-400a-9c78-52407e982478\") "
Oct 30 05:38:18.605922 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.600480    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38428a53-04c7-400a-9c78-52407e982478-tls-certs\" (UniqueName: \"kubernetes.io/secret/38428a53-04c7-400a-9c78-52407e982478-tls-certs\") pod \"38428a53-04c7-400a-9c78-52407e982478\" (UID: \"38428a53-04c7-400a-9c78-52407e982478\") "
Oct 30 05:38:18.606263 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.606063    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38428a53-04c7-400a-9c78-52407e982478-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38428a53-04c7-400a-9c78-52407e982478-tmp-dir\") pod \"38428a53-04c7-400a-9c78-52407e982478\" (UID: \"38428a53-04c7-400a-9c78-52407e982478\") "
Oct 30 05:38:18.606263 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.606116    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38428a53-04c7-400a-9c78-52407e982478-dshm\" (UniqueName: \"kubernetes.io/empty-dir/38428a53-04c7-400a-9c78-52407e982478-dshm\") pod \"38428a53-04c7-400a-9c78-52407e982478\" (UID: \"38428a53-04c7-400a-9c78-52407e982478\") "
Oct 30 05:38:18.606263 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.606226    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38428a53-04c7-400a-9c78-52407e982478-kube-api-access-zrmnz\" (UniqueName: \"kubernetes.io/projected/38428a53-04c7-400a-9c78-52407e982478-kube-api-access-zrmnz\") pod \"38428a53-04c7-400a-9c78-52407e982478\" (UID: \"38428a53-04c7-400a-9c78-52407e982478\") "
Oct 30 05:38:18.628557 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.608129    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38428a53-04c7-400a-9c78-52407e982478-tmp-dir" pod "38428a53-04c7-400a-9c78-52407e982478" (UID: "38428a53-04c7-400a-9c78-52407e982478"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:38:18.628557 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.623915    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38428a53-04c7-400a-9c78-52407e982478-home" pod "38428a53-04c7-400a-9c78-52407e982478" (UID: "38428a53-04c7-400a-9c78-52407e982478"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:38:18.628557 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.623956    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38428a53-04c7-400a-9c78-52407e982478-model-cache" pod "38428a53-04c7-400a-9c78-52407e982478" (UID: "38428a53-04c7-400a-9c78-52407e982478"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:38:18.628557 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.624223    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38428a53-04c7-400a-9c78-52407e982478-dshm" pod "38428a53-04c7-400a-9c78-52407e982478" (UID: "38428a53-04c7-400a-9c78-52407e982478"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:38:18.636488 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.636452    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38428a53-04c7-400a-9c78-52407e982478-tls-certs" pod "38428a53-04c7-400a-9c78-52407e982478" (UID: "38428a53-04c7-400a-9c78-52407e982478"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:38:18.636629 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.636576    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38428a53-04c7-400a-9c78-52407e982478-kube-api-access-zrmnz" pod "38428a53-04c7-400a-9c78-52407e982478" (UID: "38428a53-04c7-400a-9c78-52407e982478"). InnerVolumeSpecName "kube-api-access-zrmnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:38:18.637449 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.637429    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6"
Oct 30 05:38:18.651973 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.651888    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx"
Oct 30 05:38:18.708595 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.708556    5898 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38428a53-04c7-400a-9c78-52407e982478-home\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:38:18.708595 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.708594    5898 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38428a53-04c7-400a-9c78-52407e982478-model-cache\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:38:18.708595 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.708609    5898 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38428a53-04c7-400a-9c78-52407e982478-tls-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:38:18.709069 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.708650    5898 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38428a53-04c7-400a-9c78-52407e982478-tmp-dir\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:38:18.709069 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.708663    5898 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38428a53-04c7-400a-9c78-52407e982478-dshm\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:38:18.709069 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.708677    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrmnz\" (UniqueName: \"kubernetes.io/projected/38428a53-04c7-400a-9c78-52407e982478-kube-api-access-zrmnz\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:38:18.809397 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.809313    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d523008f-c740-4a41-89cb-e82e1ffa0094-home\" (UniqueName: \"kubernetes.io/empty-dir/d523008f-c740-4a41-89cb-e82e1ffa0094-home\") pod \"d523008f-c740-4a41-89cb-e82e1ffa0094\" (UID: \"d523008f-c740-4a41-89cb-e82e1ffa0094\") "
Oct 30 05:38:18.809397 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.809353    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d523008f-c740-4a41-89cb-e82e1ffa0094-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d523008f-c740-4a41-89cb-e82e1ffa0094-dshm\") pod \"d523008f-c740-4a41-89cb-e82e1ffa0094\" (UID: \"d523008f-c740-4a41-89cb-e82e1ffa0094\") "
Oct 30 05:38:18.809645 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.809414    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d523008f-c740-4a41-89cb-e82e1ffa0094-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d523008f-c740-4a41-89cb-e82e1ffa0094-tmp-dir\") pod \"d523008f-c740-4a41-89cb-e82e1ffa0094\" (UID: \"d523008f-c740-4a41-89cb-e82e1ffa0094\") "
Oct 30 05:38:18.809645 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.809461    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b09804e3-8135-4f6d-ae34-c45d6da9728e-tls-certs\" (UniqueName: \"kubernetes.io/secret/b09804e3-8135-4f6d-ae34-c45d6da9728e-tls-certs\") pod \"b09804e3-8135-4f6d-ae34-c45d6da9728e\" (UID: \"b09804e3-8135-4f6d-ae34-c45d6da9728e\") "
Oct 30 05:38:18.809645 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.809573    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d523008f-c740-4a41-89cb-e82e1ffa0094-kube-api-access-4hrhg\" (UniqueName: \"kubernetes.io/projected/d523008f-c740-4a41-89cb-e82e1ffa0094-kube-api-access-4hrhg\") pod \"d523008f-c740-4a41-89cb-e82e1ffa0094\" (UID: \"d523008f-c740-4a41-89cb-e82e1ffa0094\") "
Oct 30 05:38:18.809645 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.809603    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b09804e3-8135-4f6d-ae34-c45d6da9728e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b09804e3-8135-4f6d-ae34-c45d6da9728e-tmp-dir\") pod \"b09804e3-8135-4f6d-ae34-c45d6da9728e\" (UID: \"b09804e3-8135-4f6d-ae34-c45d6da9728e\") "
Oct 30 05:38:18.809875 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.809661    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b09804e3-8135-4f6d-ae34-c45d6da9728e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b09804e3-8135-4f6d-ae34-c45d6da9728e-model-cache\") pod \"b09804e3-8135-4f6d-ae34-c45d6da9728e\" (UID: \"b09804e3-8135-4f6d-ae34-c45d6da9728e\") "
Oct 30 05:38:18.809875 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.809695    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d523008f-c740-4a41-89cb-e82e1ffa0094-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d523008f-c740-4a41-89cb-e82e1ffa0094-model-cache\") pod \"d523008f-c740-4a41-89cb-e82e1ffa0094\" (UID: \"d523008f-c740-4a41-89cb-e82e1ffa0094\") "
Oct 30 05:38:18.809875 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.809765    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d523008f-c740-4a41-89cb-e82e1ffa0094-tls-certs\" (UniqueName: \"kubernetes.io/secret/d523008f-c740-4a41-89cb-e82e1ffa0094-tls-certs\") pod \"d523008f-c740-4a41-89cb-e82e1ffa0094\" (UID: \"d523008f-c740-4a41-89cb-e82e1ffa0094\") "
Oct 30 05:38:18.809875 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.809816    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b09804e3-8135-4f6d-ae34-c45d6da9728e-kube-api-access-m2jn9\" (UniqueName: \"kubernetes.io/projected/b09804e3-8135-4f6d-ae34-c45d6da9728e-kube-api-access-m2jn9\") pod \"b09804e3-8135-4f6d-ae34-c45d6da9728e\" (UID: \"b09804e3-8135-4f6d-ae34-c45d6da9728e\") "
Oct 30 05:38:18.810104 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.809881    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b09804e3-8135-4f6d-ae34-c45d6da9728e-home\" (UniqueName: \"kubernetes.io/empty-dir/b09804e3-8135-4f6d-ae34-c45d6da9728e-home\") pod \"b09804e3-8135-4f6d-ae34-c45d6da9728e\" (UID: \"b09804e3-8135-4f6d-ae34-c45d6da9728e\") "
Oct 30 05:38:18.810104 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.809664    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d523008f-c740-4a41-89cb-e82e1ffa0094-home" pod "d523008f-c740-4a41-89cb-e82e1ffa0094" (UID: "d523008f-c740-4a41-89cb-e82e1ffa0094"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:38:18.810104 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.809732    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d523008f-c740-4a41-89cb-e82e1ffa0094-tmp-dir" pod "d523008f-c740-4a41-89cb-e82e1ffa0094" (UID: "d523008f-c740-4a41-89cb-e82e1ffa0094"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:38:18.810104 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.809955    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b09804e3-8135-4f6d-ae34-c45d6da9728e-model-cache" pod "b09804e3-8135-4f6d-ae34-c45d6da9728e" (UID: "b09804e3-8135-4f6d-ae34-c45d6da9728e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:38:18.810104 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.809964    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d523008f-c740-4a41-89cb-e82e1ffa0094-model-cache" pod "d523008f-c740-4a41-89cb-e82e1ffa0094" (UID: "d523008f-c740-4a41-89cb-e82e1ffa0094"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:38:18.810104 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.809968    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b09804e3-8135-4f6d-ae34-c45d6da9728e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b09804e3-8135-4f6d-ae34-c45d6da9728e-dshm\") pod \"b09804e3-8135-4f6d-ae34-c45d6da9728e\" (UID: \"b09804e3-8135-4f6d-ae34-c45d6da9728e\") "
Oct 30 05:38:18.810412 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.810221    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b09804e3-8135-4f6d-ae34-c45d6da9728e-tmp-dir" pod "b09804e3-8135-4f6d-ae34-c45d6da9728e" (UID: "b09804e3-8135-4f6d-ae34-c45d6da9728e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:38:18.810492 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.810473    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b09804e3-8135-4f6d-ae34-c45d6da9728e-home" pod "b09804e3-8135-4f6d-ae34-c45d6da9728e" (UID: "b09804e3-8135-4f6d-ae34-c45d6da9728e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:38:18.810744 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.810722    5898 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b09804e3-8135-4f6d-ae34-c45d6da9728e-tmp-dir\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:38:18.810785 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.810752    5898 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b09804e3-8135-4f6d-ae34-c45d6da9728e-model-cache\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:38:18.810785 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.810766    5898 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d523008f-c740-4a41-89cb-e82e1ffa0094-model-cache\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:38:18.810785 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.810778    5898 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b09804e3-8135-4f6d-ae34-c45d6da9728e-home\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:38:18.810885 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.810791    5898 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d523008f-c740-4a41-89cb-e82e1ffa0094-home\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:38:18.810885 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.810804    5898 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d523008f-c740-4a41-89cb-e82e1ffa0094-tmp-dir\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:38:18.812210 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.812189    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d523008f-c740-4a41-89cb-e82e1ffa0094-kube-api-access-4hrhg" pod "d523008f-c740-4a41-89cb-e82e1ffa0094" (UID: "d523008f-c740-4a41-89cb-e82e1ffa0094"). InnerVolumeSpecName "kube-api-access-4hrhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:38:18.812325 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.812240    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d523008f-c740-4a41-89cb-e82e1ffa0094-dshm" pod "d523008f-c740-4a41-89cb-e82e1ffa0094" (UID: "d523008f-c740-4a41-89cb-e82e1ffa0094"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:38:18.812325 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.812247    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b09804e3-8135-4f6d-ae34-c45d6da9728e-tls-certs" pod "b09804e3-8135-4f6d-ae34-c45d6da9728e" (UID: "b09804e3-8135-4f6d-ae34-c45d6da9728e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:38:18.812325 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.812299    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b09804e3-8135-4f6d-ae34-c45d6da9728e-kube-api-access-m2jn9" pod "b09804e3-8135-4f6d-ae34-c45d6da9728e" (UID: "b09804e3-8135-4f6d-ae34-c45d6da9728e"). InnerVolumeSpecName "kube-api-access-m2jn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:38:18.812445 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.812362    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b09804e3-8135-4f6d-ae34-c45d6da9728e-dshm" pod "b09804e3-8135-4f6d-ae34-c45d6da9728e" (UID: "b09804e3-8135-4f6d-ae34-c45d6da9728e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:38:18.812656 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.812636    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d523008f-c740-4a41-89cb-e82e1ffa0094-tls-certs" pod "d523008f-c740-4a41-89cb-e82e1ffa0094" (UID: "d523008f-c740-4a41-89cb-e82e1ffa0094"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:38:18.825148 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.825122    5898 generic.go:356] "Generic (PLEG): container finished" podID="b09804e3-8135-4f6d-ae34-c45d6da9728e" containerID="28741bfcd2964321cba1ee5775c787f0ab4c32774a47338b54efd3b2d9c05fc4" exitCode=0
Oct 30 05:38:18.825265 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.825173    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6"
Oct 30 05:38:18.825265 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.825213    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6" event={"ID":"b09804e3-8135-4f6d-ae34-c45d6da9728e","Type":"ContainerDied","Data":"28741bfcd2964321cba1ee5775c787f0ab4c32774a47338b54efd3b2d9c05fc4"}
Oct 30 05:38:18.825265 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.825260    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6" event={"ID":"b09804e3-8135-4f6d-ae34-c45d6da9728e","Type":"ContainerDied","Data":"ff76c08a1188f348b2c4e99fbcc765e4d962457401ce334bc583ad4fb1aee2fe"}
Oct 30 05:38:18.825429 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.825279    5898 scope.go:122] "RemoveContainer" containerID="28741bfcd2964321cba1ee5775c787f0ab4c32774a47338b54efd3b2d9c05fc4"
Oct 30 05:38:18.826782 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.826756    5898 generic.go:356] "Generic (PLEG): container finished" podID="38428a53-04c7-400a-9c78-52407e982478" containerID="b702a116520d68edde52842852fcb463f2966912049c96ba49d8814ee70099e0" exitCode=0
Oct 30 05:38:18.826890 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.826791    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh"
Oct 30 05:38:18.826890 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.826818    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh" event={"ID":"38428a53-04c7-400a-9c78-52407e982478","Type":"ContainerDied","Data":"b702a116520d68edde52842852fcb463f2966912049c96ba49d8814ee70099e0"}
Oct 30 05:38:18.826890 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.826854    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh" event={"ID":"38428a53-04c7-400a-9c78-52407e982478","Type":"ContainerDied","Data":"9e4c9195c681f79150497445f821d0afe53ef111821571f73fc0d1f612ad1f63"}
Oct 30 05:38:18.828515 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.828490    5898 generic.go:356] "Generic (PLEG): container finished" podID="d523008f-c740-4a41-89cb-e82e1ffa0094" containerID="09527f87c929c6f43806add434b18df957d709cc967f2aba2be1e376fc61e00f" exitCode=0
Oct 30 05:38:18.828603 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.828536    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx"
Oct 30 05:38:18.828603 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.828549    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx" event={"ID":"d523008f-c740-4a41-89cb-e82e1ffa0094","Type":"ContainerDied","Data":"09527f87c929c6f43806add434b18df957d709cc967f2aba2be1e376fc61e00f"}
Oct 30 05:38:18.828603 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.828575    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx" event={"ID":"d523008f-c740-4a41-89cb-e82e1ffa0094","Type":"ContainerDied","Data":"990bb437d80bbd6999f9c10cebf9c13c1db4eeb7a2a4d714fe80601526764c7f"}
Oct 30 05:38:18.844641 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.844598    5898 scope.go:122] "RemoveContainer" containerID="28741bfcd2964321cba1ee5775c787f0ab4c32774a47338b54efd3b2d9c05fc4"
Oct 30 05:38:18.845282 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:38:18.845062    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28741bfcd2964321cba1ee5775c787f0ab4c32774a47338b54efd3b2d9c05fc4\": container with ID starting with 28741bfcd2964321cba1ee5775c787f0ab4c32774a47338b54efd3b2d9c05fc4 not found: ID does not exist" containerID="28741bfcd2964321cba1ee5775c787f0ab4c32774a47338b54efd3b2d9c05fc4"
Oct 30 05:38:18.845282 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.845109    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28741bfcd2964321cba1ee5775c787f0ab4c32774a47338b54efd3b2d9c05fc4"} err="failed to get container status \"28741bfcd2964321cba1ee5775c787f0ab4c32774a47338b54efd3b2d9c05fc4\": rpc error: code = NotFound desc = could not find container \"28741bfcd2964321cba1ee5775c787f0ab4c32774a47338b54efd3b2d9c05fc4\": container with ID starting with 28741bfcd2964321cba1ee5775c787f0ab4c32774a47338b54efd3b2d9c05fc4 not found: ID does not exist"
Oct 30 05:38:18.845282 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.845133    5898 scope.go:122] "RemoveContainer" containerID="b702a116520d68edde52842852fcb463f2966912049c96ba49d8814ee70099e0"
Oct 30 05:38:18.855085 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.855066    5898 scope.go:122] "RemoveContainer" containerID="b702a116520d68edde52842852fcb463f2966912049c96ba49d8814ee70099e0"
Oct 30 05:38:18.855383 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:38:18.855364    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b702a116520d68edde52842852fcb463f2966912049c96ba49d8814ee70099e0\": container with ID starting with b702a116520d68edde52842852fcb463f2966912049c96ba49d8814ee70099e0 not found: ID does not exist" containerID="b702a116520d68edde52842852fcb463f2966912049c96ba49d8814ee70099e0"
Oct 30 05:38:18.855442 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.855391    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b702a116520d68edde52842852fcb463f2966912049c96ba49d8814ee70099e0"} err="failed to get container status \"b702a116520d68edde52842852fcb463f2966912049c96ba49d8814ee70099e0\": rpc error: code = NotFound desc = could not find container \"b702a116520d68edde52842852fcb463f2966912049c96ba49d8814ee70099e0\": container with ID starting with b702a116520d68edde52842852fcb463f2966912049c96ba49d8814ee70099e0 not found: ID does not exist"
Oct 30 05:38:18.855442 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.855424    5898 scope.go:122] "RemoveContainer" containerID="09527f87c929c6f43806add434b18df957d709cc967f2aba2be1e376fc61e00f"
Oct 30 05:38:18.863830 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.863810    5898 scope.go:122] "RemoveContainer" containerID="09527f87c929c6f43806add434b18df957d709cc967f2aba2be1e376fc61e00f"
Oct 30 05:38:18.864115 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:38:18.864096    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09527f87c929c6f43806add434b18df957d709cc967f2aba2be1e376fc61e00f\": container with ID starting with 09527f87c929c6f43806add434b18df957d709cc967f2aba2be1e376fc61e00f not found: ID does not exist" containerID="09527f87c929c6f43806add434b18df957d709cc967f2aba2be1e376fc61e00f"
Oct 30 05:38:18.864188 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.864122    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09527f87c929c6f43806add434b18df957d709cc967f2aba2be1e376fc61e00f"} err="failed to get container status \"09527f87c929c6f43806add434b18df957d709cc967f2aba2be1e376fc61e00f\": rpc error: code = NotFound desc = could not find container \"09527f87c929c6f43806add434b18df957d709cc967f2aba2be1e376fc61e00f\": container with ID starting with 09527f87c929c6f43806add434b18df957d709cc967f2aba2be1e376fc61e00f not found: ID does not exist"
Oct 30 05:38:18.910237 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.910210    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6"]
Oct 30 05:38:18.911618 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.911593    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2jn9\" (UniqueName: \"kubernetes.io/projected/b09804e3-8135-4f6d-ae34-c45d6da9728e-kube-api-access-m2jn9\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:38:18.912126 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.911621    5898 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b09804e3-8135-4f6d-ae34-c45d6da9728e-dshm\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:38:18.912201 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.912136    5898 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d523008f-c740-4a41-89cb-e82e1ffa0094-dshm\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:38:18.912201 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.912146    5898 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b09804e3-8135-4f6d-ae34-c45d6da9728e-tls-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:38:18.912201 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.912156    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hrhg\" (UniqueName: \"kubernetes.io/projected/d523008f-c740-4a41-89cb-e82e1ffa0094-kube-api-access-4hrhg\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:38:18.912201 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.912169    5898 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d523008f-c740-4a41-89cb-e82e1ffa0094-tls-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:38:18.914716 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.914693    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-b21507/distinct-w0-b21507-kserve-866698fd8b-hjkp6"]
Oct 30 05:38:18.977710 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.977678    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx"]
Oct 30 05:38:18.994478 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:18.994438    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-b21507/simulator-w0-b21507-kserve-5f4f779998-7msgx"]
Oct 30 05:38:19.030190 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:19.030160    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh"]
Oct 30 05:38:19.041808 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:19.038929    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-b21507/premium-w0-b21507-kserve-75db566f55-sxwvh"]
Oct 30 05:38:19.046485 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:19.046456    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:38:19.245605 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:19.245577    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:20.890711 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:20.890671    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38428a53-04c7-400a-9c78-52407e982478" path="/var/lib/kubelet/pods/38428a53-04c7-400a-9c78-52407e982478/volumes"
Oct 30 05:38:20.891341 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:20.891319    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b09804e3-8135-4f6d-ae34-c45d6da9728e" path="/var/lib/kubelet/pods/b09804e3-8135-4f6d-ae34-c45d6da9728e/volumes"
Oct 30 05:38:20.891806 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:20.891785    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d523008f-c740-4a41-89cb-e82e1ffa0094" path="/var/lib/kubelet/pods/d523008f-c740-4a41-89cb-e82e1ffa0094/volumes"
Oct 30 05:38:21.765168 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:21.765135    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:22.135118 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:22.135075    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-b21507-59b47dc888-29s5k" podUID="30460201-04af-49fb-a7d3-5ee121e3bdb4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:38:22.135118 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:22.135107    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b21507-b7c9cc4c5-bkrt5" podUID="c00385be-b173-422b-8ba6-0985c9cf7598" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:38:23.777839 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:23.777805    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:25.635627 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:25.635335    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:26.818945 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:38:26.818892    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:38:27.554677 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:27.554643    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:29.218798 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:29.218763    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-93bad9-54ff46887b-kzztl"]
Oct 30 05:38:29.219329 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:29.219001    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-93bad9-54ff46887b-kzztl" podUID="f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc" containerName="maas-api" containerID="cri-o://33c6f507394bef4f9461f01a748b74f4477a01a6d8dceb4b9f964a623f006cb7" gracePeriod=30
Oct 30 05:38:29.249230 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:29.249198    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:29.370889 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:29.370851    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:29.540021 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:29.539950    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-93bad9-54ff46887b-kzztl"
Oct 30 05:38:29.642980 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:29.642946    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc-maas-api-tls\") pod \"f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc\" (UID: \"f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc\") "
Oct 30 05:38:29.643176 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:29.643015    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc-kube-api-access-t4ss2\" (UniqueName: \"kubernetes.io/projected/f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc-kube-api-access-t4ss2\") pod \"f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc\" (UID: \"f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc\") "
Oct 30 05:38:29.643176 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:29.643080    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc-metrics-certs\") pod \"f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc\" (UID: \"f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc\") "
Oct 30 05:38:29.643310 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:29.643253    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc-openshift-service-ca\") pod \"f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc\" (UID: \"f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc\") "
Oct 30 05:38:29.643654 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:29.643627    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc-openshift-service-ca" pod "f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc" (UID: "f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:38:29.645460 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:29.645426    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc-kube-api-access-t4ss2" pod "f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc" (UID: "f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc"). InnerVolumeSpecName "kube-api-access-t4ss2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:38:29.646037 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:29.646012    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc-metrics-certs" pod "f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc" (UID: "f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:38:29.646179 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:29.646159    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc-maas-api-tls" pod "f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc" (UID: "f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:38:29.745080 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:29.744991    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4ss2\" (UniqueName: \"kubernetes.io/projected/f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc-kube-api-access-t4ss2\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:38:29.745318 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:29.745299    5898 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc-metrics-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:38:29.745507 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:29.745491    5898 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc-openshift-service-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:38:29.745619 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:29.745605    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc-maas-api-tls\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:38:29.845478 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:29.845445    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:29.903713 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:29.903677    5898 generic.go:356] "Generic (PLEG): container finished" podID="f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc" containerID="33c6f507394bef4f9461f01a748b74f4477a01a6d8dceb4b9f964a623f006cb7" exitCode=0
Oct 30 05:38:29.903937 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:29.903725    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-93bad9-54ff46887b-kzztl"
Oct 30 05:38:29.903937 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:29.903763    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-93bad9-54ff46887b-kzztl" event={"ID":"f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc","Type":"ContainerDied","Data":"33c6f507394bef4f9461f01a748b74f4477a01a6d8dceb4b9f964a623f006cb7"}
Oct 30 05:38:29.903937 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:29.903830    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-93bad9-54ff46887b-kzztl" event={"ID":"f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc","Type":"ContainerDied","Data":"91fea727d1a6abc5ea1ac24e0ec12f2667ccc77e0efe645129173ad5ba29c190"}
Oct 30 05:38:29.903937 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:29.903862    5898 scope.go:122] "RemoveContainer" containerID="33c6f507394bef4f9461f01a748b74f4477a01a6d8dceb4b9f964a623f006cb7"
Oct 30 05:38:29.914794 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:29.914775    5898 scope.go:122] "RemoveContainer" containerID="33c6f507394bef4f9461f01a748b74f4477a01a6d8dceb4b9f964a623f006cb7"
Oct 30 05:38:29.915082 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:38:29.915063    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33c6f507394bef4f9461f01a748b74f4477a01a6d8dceb4b9f964a623f006cb7\": container with ID starting with 33c6f507394bef4f9461f01a748b74f4477a01a6d8dceb4b9f964a623f006cb7 not found: ID does not exist" containerID="33c6f507394bef4f9461f01a748b74f4477a01a6d8dceb4b9f964a623f006cb7"
Oct 30 05:38:29.915142 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:29.915091    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33c6f507394bef4f9461f01a748b74f4477a01a6d8dceb4b9f964a623f006cb7"} err="failed to get container status \"33c6f507394bef4f9461f01a748b74f4477a01a6d8dceb4b9f964a623f006cb7\": rpc error: code = NotFound desc = could not find container \"33c6f507394bef4f9461f01a748b74f4477a01a6d8dceb4b9f964a623f006cb7\": container with ID starting with 33c6f507394bef4f9461f01a748b74f4477a01a6d8dceb4b9f964a623f006cb7 not found: ID does not exist"
Oct 30 05:38:29.958057 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:29.958025    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-93bad9-54ff46887b-kzztl"]
Oct 30 05:38:29.971643 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:29.971613    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-93bad9-54ff46887b-kzztl"]
Oct 30 05:38:29.974076 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:29.974047    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:30.076936 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:30.076872    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:30.514188 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:30.514093    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:30.890985 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:30.890954    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc" path="/var/lib/kubelet/pods/f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc/volumes"
Oct 30 05:38:32.135079 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:32.135046    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-b21507-59b47dc888-29s5k" podUID="30460201-04af-49fb-a7d3-5ee121e3bdb4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:38:32.135079 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:32.135064    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b21507-b7c9cc4c5-bkrt5" podUID="c00385be-b173-422b-8ba6-0985c9cf7598" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:38:32.529616 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:32.529530    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:33.432884 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:33.432856    5898 scope.go:122] "RemoveContainer" containerID="fe5cb84299f6f28a53af1b34ffaeb2efa1ab7078dd6849ef371f0abfb25f9a1b"
Oct 30 05:38:34.109814 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:34.109790    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-b21507-59b47dc888-29s5k"
Oct 30 05:38:34.141840 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:34.141816    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b21507-b7c9cc4c5-bkrt5"
Oct 30 05:38:34.205155 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:34.205121    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30460201-04af-49fb-a7d3-5ee121e3bdb4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30460201-04af-49fb-a7d3-5ee121e3bdb4-plugins-config-volume\") pod \"30460201-04af-49fb-a7d3-5ee121e3bdb4\" (UID: \"30460201-04af-49fb-a7d3-5ee121e3bdb4\") "
Oct 30 05:38:34.205338 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:34.205268    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30460201-04af-49fb-a7d3-5ee121e3bdb4-kube-api-access-mznvr\" (UniqueName: \"kubernetes.io/projected/30460201-04af-49fb-a7d3-5ee121e3bdb4-kube-api-access-mznvr\") pod \"30460201-04af-49fb-a7d3-5ee121e3bdb4\" (UID: \"30460201-04af-49fb-a7d3-5ee121e3bdb4\") "
Oct 30 05:38:34.205338 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:34.205300    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c00385be-b173-422b-8ba6-0985c9cf7598-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c00385be-b173-422b-8ba6-0985c9cf7598-plugins-config-volume\") pod \"c00385be-b173-422b-8ba6-0985c9cf7598\" (UID: \"c00385be-b173-422b-8ba6-0985c9cf7598\") "
Oct 30 05:38:34.205459 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:34.205444    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c00385be-b173-422b-8ba6-0985c9cf7598-kube-api-access-v5zlr\" (UniqueName: \"kubernetes.io/projected/c00385be-b173-422b-8ba6-0985c9cf7598-kube-api-access-v5zlr\") pod \"c00385be-b173-422b-8ba6-0985c9cf7598\" (UID: \"c00385be-b173-422b-8ba6-0985c9cf7598\") "
Oct 30 05:38:34.205747 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:34.205711    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30460201-04af-49fb-a7d3-5ee121e3bdb4-plugins-config-volume" pod "30460201-04af-49fb-a7d3-5ee121e3bdb4" (UID: "30460201-04af-49fb-a7d3-5ee121e3bdb4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:38:34.205747 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:34.205724    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c00385be-b173-422b-8ba6-0985c9cf7598-plugins-config-volume" pod "c00385be-b173-422b-8ba6-0985c9cf7598" (UID: "c00385be-b173-422b-8ba6-0985c9cf7598"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:38:34.205990 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:34.205966    5898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c00385be-b173-422b-8ba6-0985c9cf7598-plugins-config-volume\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:38:34.206069 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:34.205989    5898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30460201-04af-49fb-a7d3-5ee121e3bdb4-plugins-config-volume\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:38:34.207602 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:34.207571    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30460201-04af-49fb-a7d3-5ee121e3bdb4-kube-api-access-mznvr" pod "30460201-04af-49fb-a7d3-5ee121e3bdb4" (UID: "30460201-04af-49fb-a7d3-5ee121e3bdb4"). InnerVolumeSpecName "kube-api-access-mznvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:38:34.207707 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:34.207618    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c00385be-b173-422b-8ba6-0985c9cf7598-kube-api-access-v5zlr" pod "c00385be-b173-422b-8ba6-0985c9cf7598" (UID: "c00385be-b173-422b-8ba6-0985c9cf7598"). InnerVolumeSpecName "kube-api-access-v5zlr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:38:34.306824 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:34.306794    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mznvr\" (UniqueName: \"kubernetes.io/projected/30460201-04af-49fb-a7d3-5ee121e3bdb4-kube-api-access-mznvr\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:38:34.306824 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:34.306822    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5zlr\" (UniqueName: \"kubernetes.io/projected/c00385be-b173-422b-8ba6-0985c9cf7598-kube-api-access-v5zlr\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:38:34.935563 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:34.935522    5898 generic.go:356] "Generic (PLEG): container finished" podID="c00385be-b173-422b-8ba6-0985c9cf7598" containerID="b37375a2fb7eaaf4020e964ad8c46fb6f2cee83c7c4ab28a874c230648b93568" exitCode=0
Oct 30 05:38:34.935983 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:34.935580    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b21507-b7c9cc4c5-bkrt5"
Oct 30 05:38:34.935983 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:34.935600    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b21507-b7c9cc4c5-bkrt5" event={"ID":"c00385be-b173-422b-8ba6-0985c9cf7598","Type":"ContainerDied","Data":"b37375a2fb7eaaf4020e964ad8c46fb6f2cee83c7c4ab28a874c230648b93568"}
Oct 30 05:38:34.935983 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:34.935680    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b21507-b7c9cc4c5-bkrt5" event={"ID":"c00385be-b173-422b-8ba6-0985c9cf7598","Type":"ContainerDied","Data":"896b5d7594672fd4df164a73bcfafd39a7fdcee6f159f7dedbcf1d5ed85c5c30"}
Oct 30 05:38:34.935983 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:34.935712    5898 scope.go:122] "RemoveContainer" containerID="b37375a2fb7eaaf4020e964ad8c46fb6f2cee83c7c4ab28a874c230648b93568"
Oct 30 05:38:34.937069 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:34.937048    5898 generic.go:356] "Generic (PLEG): container finished" podID="30460201-04af-49fb-a7d3-5ee121e3bdb4" containerID="4ba9604e0c21f1b6d817fe4ca8f0099336f3de54d1b09c6a9ac0170423555310" exitCode=0
Oct 30 05:38:34.937176 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:34.937109    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-b21507-59b47dc888-29s5k"
Oct 30 05:38:34.937176 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:34.937132    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-b21507-59b47dc888-29s5k" event={"ID":"30460201-04af-49fb-a7d3-5ee121e3bdb4","Type":"ContainerDied","Data":"4ba9604e0c21f1b6d817fe4ca8f0099336f3de54d1b09c6a9ac0170423555310"}
Oct 30 05:38:34.937176 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:34.937171    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-b21507-59b47dc888-29s5k" event={"ID":"30460201-04af-49fb-a7d3-5ee121e3bdb4","Type":"ContainerDied","Data":"547b195778b359aafee4133965662df0392a839ab42592baaa1d40e70ed49172"}
Oct 30 05:38:34.945263 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:34.945243    5898 scope.go:122] "RemoveContainer" containerID="b37375a2fb7eaaf4020e964ad8c46fb6f2cee83c7c4ab28a874c230648b93568"
Oct 30 05:38:34.945544 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:38:34.945525    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b37375a2fb7eaaf4020e964ad8c46fb6f2cee83c7c4ab28a874c230648b93568\": container with ID starting with b37375a2fb7eaaf4020e964ad8c46fb6f2cee83c7c4ab28a874c230648b93568 not found: ID does not exist" containerID="b37375a2fb7eaaf4020e964ad8c46fb6f2cee83c7c4ab28a874c230648b93568"
Oct 30 05:38:34.945624 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:34.945555    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b37375a2fb7eaaf4020e964ad8c46fb6f2cee83c7c4ab28a874c230648b93568"} err="failed to get container status \"b37375a2fb7eaaf4020e964ad8c46fb6f2cee83c7c4ab28a874c230648b93568\": rpc error: code = NotFound desc = could not find container \"b37375a2fb7eaaf4020e964ad8c46fb6f2cee83c7c4ab28a874c230648b93568\": container with ID starting with b37375a2fb7eaaf4020e964ad8c46fb6f2cee83c7c4ab28a874c230648b93568 not found: ID does not exist"
Oct 30 05:38:34.945624 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:34.945579    5898 scope.go:122] "RemoveContainer" containerID="4ba9604e0c21f1b6d817fe4ca8f0099336f3de54d1b09c6a9ac0170423555310"
Oct 30 05:38:34.954068 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:34.954050    5898 scope.go:122] "RemoveContainer" containerID="4ba9604e0c21f1b6d817fe4ca8f0099336f3de54d1b09c6a9ac0170423555310"
Oct 30 05:38:34.954324 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:38:34.954306    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ba9604e0c21f1b6d817fe4ca8f0099336f3de54d1b09c6a9ac0170423555310\": container with ID starting with 4ba9604e0c21f1b6d817fe4ca8f0099336f3de54d1b09c6a9ac0170423555310 not found: ID does not exist" containerID="4ba9604e0c21f1b6d817fe4ca8f0099336f3de54d1b09c6a9ac0170423555310"
Oct 30 05:38:34.954389 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:34.954334    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ba9604e0c21f1b6d817fe4ca8f0099336f3de54d1b09c6a9ac0170423555310"} err="failed to get container status \"4ba9604e0c21f1b6d817fe4ca8f0099336f3de54d1b09c6a9ac0170423555310\": rpc error: code = NotFound desc = could not find container \"4ba9604e0c21f1b6d817fe4ca8f0099336f3de54d1b09c6a9ac0170423555310\": container with ID starting with 4ba9604e0c21f1b6d817fe4ca8f0099336f3de54d1b09c6a9ac0170423555310 not found: ID does not exist"
Oct 30 05:38:34.987092 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:34.987062    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b21507-b7c9cc4c5-bkrt5"]
Oct 30 05:38:34.997844 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:34.997825    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b21507-b7c9cc4c5-bkrt5"]
Oct 30 05:38:35.022569 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:35.022546    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b21507-59b47dc888-29s5k"]
Oct 30 05:38:35.026555 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:35.026536    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b21507-59b47dc888-29s5k"]
Oct 30 05:38:35.389352 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:35.389322    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:36.892138 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:36.892106    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30460201-04af-49fb-a7d3-5ee121e3bdb4" path="/var/lib/kubelet/pods/30460201-04af-49fb-a7d3-5ee121e3bdb4/volumes"
Oct 30 05:38:36.894613 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:36.892589    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c00385be-b173-422b-8ba6-0985c9cf7598" path="/var/lib/kubelet/pods/c00385be-b173-422b-8ba6-0985c9cf7598/volumes"
Oct 30 05:38:37.177325 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:37.177217    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:37.329559 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:37.329523    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:37.580003 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:37.579973    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5"]
Oct 30 05:38:37.580244 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:37.580214    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5" podUID="59cf2966-7359-48ba-bd83-769f0c076397" containerName="istio-proxy" containerID="cri-o://db826a7fc82a89847819d280211b43bac38451cf91ea593a52b6faaac97fa63d" gracePeriod=30
Oct 30 05:38:37.612016 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:37.611989    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:38.054647 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:38.054565    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:39.396188 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:39.396142    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:40.286052 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:40.286015    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:40.409355 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:40.409317    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:40.786748 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:40.786716    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:38:41.151184 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:41.151153    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:41.356359 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:41.356330    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:41.524025 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:41.523945    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:41.870870 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:41.863539    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:42.043253 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:42.043176    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:42.282002 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:42.280919    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:42.599130 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:42.599106    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:42.764750 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:42.764720    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:42.848592 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:42.848537    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5"
Oct 30 05:38:42.874894 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:42.874863    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59cf2966-7359-48ba-bd83-769f0c076397-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59cf2966-7359-48ba-bd83-769f0c076397-istiod-ca-cert\") pod \"59cf2966-7359-48ba-bd83-769f0c076397\" (UID: \"59cf2966-7359-48ba-bd83-769f0c076397\") "
Oct 30 05:38:42.875071 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:42.874936    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59cf2966-7359-48ba-bd83-769f0c076397-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59cf2966-7359-48ba-bd83-769f0c076397-workload-socket\") pod \"59cf2966-7359-48ba-bd83-769f0c076397\" (UID: \"59cf2966-7359-48ba-bd83-769f0c076397\") "
Oct 30 05:38:42.875071 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:42.874997    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59cf2966-7359-48ba-bd83-769f0c076397-istio-token\" (UniqueName: \"kubernetes.io/projected/59cf2966-7359-48ba-bd83-769f0c076397-istio-token\") pod \"59cf2966-7359-48ba-bd83-769f0c076397\" (UID: \"59cf2966-7359-48ba-bd83-769f0c076397\") "
Oct 30 05:38:42.875219 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:42.875104    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59cf2966-7359-48ba-bd83-769f0c076397-kube-api-access-45nn7\" (UniqueName: \"kubernetes.io/projected/59cf2966-7359-48ba-bd83-769f0c076397-kube-api-access-45nn7\") pod \"59cf2966-7359-48ba-bd83-769f0c076397\" (UID: \"59cf2966-7359-48ba-bd83-769f0c076397\") "
Oct 30 05:38:42.875219 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:42.875134    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59cf2966-7359-48ba-bd83-769f0c076397-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59cf2966-7359-48ba-bd83-769f0c076397-credential-socket\") pod \"59cf2966-7359-48ba-bd83-769f0c076397\" (UID: \"59cf2966-7359-48ba-bd83-769f0c076397\") "
Oct 30 05:38:42.875219 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:42.875162    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59cf2966-7359-48ba-bd83-769f0c076397-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59cf2966-7359-48ba-bd83-769f0c076397-istio-data\") pod \"59cf2966-7359-48ba-bd83-769f0c076397\" (UID: \"59cf2966-7359-48ba-bd83-769f0c076397\") "
Oct 30 05:38:42.875392 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:42.875223    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/59cf2966-7359-48ba-bd83-769f0c076397-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59cf2966-7359-48ba-bd83-769f0c076397-istio-podinfo\") pod \"59cf2966-7359-48ba-bd83-769f0c076397\" (UID: \"59cf2966-7359-48ba-bd83-769f0c076397\") "
Oct 30 05:38:42.875392 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:42.875257    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59cf2966-7359-48ba-bd83-769f0c076397-workload-socket" pod "59cf2966-7359-48ba-bd83-769f0c076397" (UID: "59cf2966-7359-48ba-bd83-769f0c076397"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:38:42.875392 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:42.875312    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59cf2966-7359-48ba-bd83-769f0c076397-istiod-ca-cert" pod "59cf2966-7359-48ba-bd83-769f0c076397" (UID: "59cf2966-7359-48ba-bd83-769f0c076397"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:38:42.875570 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:42.875396    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59cf2966-7359-48ba-bd83-769f0c076397-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59cf2966-7359-48ba-bd83-769f0c076397-workload-certs\") pod \"59cf2966-7359-48ba-bd83-769f0c076397\" (UID: \"59cf2966-7359-48ba-bd83-769f0c076397\") "
Oct 30 05:38:42.875570 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:42.875433    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59cf2966-7359-48ba-bd83-769f0c076397-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59cf2966-7359-48ba-bd83-769f0c076397-istio-envoy\") pod \"59cf2966-7359-48ba-bd83-769f0c076397\" (UID: \"59cf2966-7359-48ba-bd83-769f0c076397\") "
Oct 30 05:38:42.875570 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:42.875483    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59cf2966-7359-48ba-bd83-769f0c076397-credential-socket" pod "59cf2966-7359-48ba-bd83-769f0c076397" (UID: "59cf2966-7359-48ba-bd83-769f0c076397"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:38:42.875738 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:42.875627    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59cf2966-7359-48ba-bd83-769f0c076397-istio-data" pod "59cf2966-7359-48ba-bd83-769f0c076397" (UID: "59cf2966-7359-48ba-bd83-769f0c076397"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:38:42.875738 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:42.875724    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59cf2966-7359-48ba-bd83-769f0c076397-workload-certs" pod "59cf2966-7359-48ba-bd83-769f0c076397" (UID: "59cf2966-7359-48ba-bd83-769f0c076397"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:38:42.876038 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:42.876017    5898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59cf2966-7359-48ba-bd83-769f0c076397-workload-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:38:42.876149 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:42.876043    5898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59cf2966-7359-48ba-bd83-769f0c076397-istiod-ca-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:38:42.876149 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:42.876057    5898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59cf2966-7359-48ba-bd83-769f0c076397-workload-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:38:42.876149 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:42.876073    5898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59cf2966-7359-48ba-bd83-769f0c076397-credential-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:38:42.876149 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:42.876089    5898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59cf2966-7359-48ba-bd83-769f0c076397-istio-data\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:38:42.877489 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:42.877467    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59cf2966-7359-48ba-bd83-769f0c076397-istio-token" pod "59cf2966-7359-48ba-bd83-769f0c076397" (UID: "59cf2966-7359-48ba-bd83-769f0c076397"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:38:42.877489 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:42.877466    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59cf2966-7359-48ba-bd83-769f0c076397-istio-podinfo" pod "59cf2966-7359-48ba-bd83-769f0c076397" (UID: "59cf2966-7359-48ba-bd83-769f0c076397"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:38:42.877670 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:42.877625    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59cf2966-7359-48ba-bd83-769f0c076397-istio-envoy" pod "59cf2966-7359-48ba-bd83-769f0c076397" (UID: "59cf2966-7359-48ba-bd83-769f0c076397"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:38:42.877884 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:42.877868    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59cf2966-7359-48ba-bd83-769f0c076397-kube-api-access-45nn7" pod "59cf2966-7359-48ba-bd83-769f0c076397" (UID: "59cf2966-7359-48ba-bd83-769f0c076397"). InnerVolumeSpecName "kube-api-access-45nn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:38:42.896321 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:42.896294    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:42.984551 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:42.984508    5898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59cf2966-7359-48ba-bd83-769f0c076397-istio-envoy\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:38:42.984551 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:42.984548    5898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59cf2966-7359-48ba-bd83-769f0c076397-istio-token\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:38:42.984850 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:42.984564    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45nn7\" (UniqueName: \"kubernetes.io/projected/59cf2966-7359-48ba-bd83-769f0c076397-kube-api-access-45nn7\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:38:42.984850 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:42.984579    5898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59cf2966-7359-48ba-bd83-769f0c076397-istio-podinfo\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:38:43.012399 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:43.012360    5898 generic.go:356] "Generic (PLEG): container finished" podID="59cf2966-7359-48ba-bd83-769f0c076397" containerID="db826a7fc82a89847819d280211b43bac38451cf91ea593a52b6faaac97fa63d" exitCode=0
Oct 30 05:38:43.012622 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:43.012406    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5"
Oct 30 05:38:43.012622 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:43.012446    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5" event={"ID":"59cf2966-7359-48ba-bd83-769f0c076397","Type":"ContainerDied","Data":"db826a7fc82a89847819d280211b43bac38451cf91ea593a52b6faaac97fa63d"}
Oct 30 05:38:43.012622 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:43.012504    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5" event={"ID":"59cf2966-7359-48ba-bd83-769f0c076397","Type":"ContainerDied","Data":"f18d6510e05120b9121fd0842ba7bdf47ef99bb2d3803e3ea92c49e737235e22"}
Oct 30 05:38:43.012622 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:43.012530    5898 scope.go:122] "RemoveContainer" containerID="db826a7fc82a89847819d280211b43bac38451cf91ea593a52b6faaac97fa63d"
Oct 30 05:38:43.022851 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:43.022830    5898 scope.go:122] "RemoveContainer" containerID="db826a7fc82a89847819d280211b43bac38451cf91ea593a52b6faaac97fa63d"
Oct 30 05:38:43.023164 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:38:43.023141    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db826a7fc82a89847819d280211b43bac38451cf91ea593a52b6faaac97fa63d\": container with ID starting with db826a7fc82a89847819d280211b43bac38451cf91ea593a52b6faaac97fa63d not found: ID does not exist" containerID="db826a7fc82a89847819d280211b43bac38451cf91ea593a52b6faaac97fa63d"
Oct 30 05:38:43.023241 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:43.023177    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db826a7fc82a89847819d280211b43bac38451cf91ea593a52b6faaac97fa63d"} err="failed to get container status \"db826a7fc82a89847819d280211b43bac38451cf91ea593a52b6faaac97fa63d\": rpc error: code = NotFound desc = could not find container \"db826a7fc82a89847819d280211b43bac38451cf91ea593a52b6faaac97fa63d\": container with ID starting with db826a7fc82a89847819d280211b43bac38451cf91ea593a52b6faaac97fa63d not found: ID does not exist"
Oct 30 05:38:43.056137 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:43.056108    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:43.066875 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:43.066851    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6"]
Oct 30 05:38:43.078300 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:43.078275    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb"]
Oct 30 05:38:43.108008 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:43.107892    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49"]
Oct 30 05:38:43.110647 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:43.110620    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5"]
Oct 30 05:38:43.116787 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:43.116762    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-93bad9-openshift-default-67d47f87d9-h7kq5"]
Oct 30 05:38:43.256754 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:43.256714    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:43.472530 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:43.472427    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:43.887168 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:43.887134    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:44.090894 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:44.090861    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:44.897027 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:44.896984    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59cf2966-7359-48ba-bd83-769f0c076397" path="/var/lib/kubelet/pods/59cf2966-7359-48ba-bd83-769f0c076397/volumes"
Oct 30 05:38:45.329074 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:45.329043    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:47.294353 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:47.294321    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:48.916360 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:48.916326    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:49.037965 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:49.037932    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:49.376006 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:49.375972    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:49.972883 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:49.972847    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:50.088927 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:50.087944    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:50.274490 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:50.274389    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:51.341605 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:51.341575    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:51.782920 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:51.782788    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:51.937480 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:51.937445    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:52.183359 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:52.183293    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:52.673060 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:52.673023    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:52.811595 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:52.811536    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:53.575417 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:53.575381    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:56.021789 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:56.021753    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:56.646186 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:56.646157    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:56.776963 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:56.776930    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:56.853926 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:38:56.852483    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:38:56.900980 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:56.900855    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:57.399444 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:57.399412    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:58.089946 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:58.089866    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6" podUID="47ff272a-ac5c-4f59-bf7b-5c0692d09386" containerName="main" containerID="cri-o://5ea897a4d77a647b44003d22b196cc3136b848404e7f753cb8fd5d8a9a4bd7c8" gracePeriod=45
Oct 30 05:38:58.102682 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:58.102646    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb" podUID="6f16bd6a-a7f0-49bb-844f-57b777bde2ed" containerName="main" containerID="cri-o://9e43b28fab1725efc5f8953ba3c552342e7224362b718c643f783f26beb042b2" gracePeriod=45
Oct 30 05:38:58.129500 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:58.129439    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49" podUID="05eef4eb-f379-4dc7-a947-123bd85a4968" containerName="main" containerID="cri-o://4f15f34f28e86ff388667b9f6ae2d0c86f1c6931168dfb4f72443c054797aef8" gracePeriod=45
Oct 30 05:38:58.417343 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:58.417318    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb"
Oct 30 05:38:58.420050 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:58.420019    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6"
Oct 30 05:38:58.461184 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:58.461160    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49"
Oct 30 05:38:58.597020 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:58.596989    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f16bd6a-a7f0-49bb-844f-57b777bde2ed-tls-certs\" (UniqueName: \"kubernetes.io/secret/6f16bd6a-a7f0-49bb-844f-57b777bde2ed-tls-certs\") pod \"6f16bd6a-a7f0-49bb-844f-57b777bde2ed\" (UID: \"6f16bd6a-a7f0-49bb-844f-57b777bde2ed\") "
Oct 30 05:38:58.597335 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:58.597084    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05eef4eb-f379-4dc7-a947-123bd85a4968-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05eef4eb-f379-4dc7-a947-123bd85a4968-model-cache\") pod \"05eef4eb-f379-4dc7-a947-123bd85a4968\" (UID: \"05eef4eb-f379-4dc7-a947-123bd85a4968\") "
Oct 30 05:38:58.597335 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:58.597134    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f16bd6a-a7f0-49bb-844f-57b777bde2ed-kube-api-access-2j75z\" (UniqueName: \"kubernetes.io/projected/6f16bd6a-a7f0-49bb-844f-57b777bde2ed-kube-api-access-2j75z\") pod \"6f16bd6a-a7f0-49bb-844f-57b777bde2ed\" (UID: \"6f16bd6a-a7f0-49bb-844f-57b777bde2ed\") "
Oct 30 05:38:58.597335 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:58.597175    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47ff272a-ac5c-4f59-bf7b-5c0692d09386-tls-certs\" (UniqueName: \"kubernetes.io/secret/47ff272a-ac5c-4f59-bf7b-5c0692d09386-tls-certs\") pod \"47ff272a-ac5c-4f59-bf7b-5c0692d09386\" (UID: \"47ff272a-ac5c-4f59-bf7b-5c0692d09386\") "
Oct 30 05:38:58.597335 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:58.597236    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05eef4eb-f379-4dc7-a947-123bd85a4968-home\" (UniqueName: \"kubernetes.io/empty-dir/05eef4eb-f379-4dc7-a947-123bd85a4968-home\") pod \"05eef4eb-f379-4dc7-a947-123bd85a4968\" (UID: \"05eef4eb-f379-4dc7-a947-123bd85a4968\") "
Oct 30 05:38:58.597335 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:58.597276    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47ff272a-ac5c-4f59-bf7b-5c0692d09386-home\" (UniqueName: \"kubernetes.io/empty-dir/47ff272a-ac5c-4f59-bf7b-5c0692d09386-home\") pod \"47ff272a-ac5c-4f59-bf7b-5c0692d09386\" (UID: \"47ff272a-ac5c-4f59-bf7b-5c0692d09386\") "
Oct 30 05:38:58.597335 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:58.597333    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05eef4eb-f379-4dc7-a947-123bd85a4968-kube-api-access-rtpk5\" (UniqueName: \"kubernetes.io/projected/05eef4eb-f379-4dc7-a947-123bd85a4968-kube-api-access-rtpk5\") pod \"05eef4eb-f379-4dc7-a947-123bd85a4968\" (UID: \"05eef4eb-f379-4dc7-a947-123bd85a4968\") "
Oct 30 05:38:58.597700 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:58.597340    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05eef4eb-f379-4dc7-a947-123bd85a4968-model-cache" pod "05eef4eb-f379-4dc7-a947-123bd85a4968" (UID: "05eef4eb-f379-4dc7-a947-123bd85a4968"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:38:58.597700 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:58.597379    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05eef4eb-f379-4dc7-a947-123bd85a4968-tls-certs\" (UniqueName: \"kubernetes.io/secret/05eef4eb-f379-4dc7-a947-123bd85a4968-tls-certs\") pod \"05eef4eb-f379-4dc7-a947-123bd85a4968\" (UID: \"05eef4eb-f379-4dc7-a947-123bd85a4968\") "
Oct 30 05:38:58.597700 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:58.597431    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f16bd6a-a7f0-49bb-844f-57b777bde2ed-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f16bd6a-a7f0-49bb-844f-57b777bde2ed-dshm\") pod \"6f16bd6a-a7f0-49bb-844f-57b777bde2ed\" (UID: \"6f16bd6a-a7f0-49bb-844f-57b777bde2ed\") "
Oct 30 05:38:58.597700 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:58.597456    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f16bd6a-a7f0-49bb-844f-57b777bde2ed-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f16bd6a-a7f0-49bb-844f-57b777bde2ed-model-cache\") pod \"6f16bd6a-a7f0-49bb-844f-57b777bde2ed\" (UID: \"6f16bd6a-a7f0-49bb-844f-57b777bde2ed\") "
Oct 30 05:38:58.597700 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:58.597503    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f16bd6a-a7f0-49bb-844f-57b777bde2ed-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f16bd6a-a7f0-49bb-844f-57b777bde2ed-tmp-dir\") pod \"6f16bd6a-a7f0-49bb-844f-57b777bde2ed\" (UID: \"6f16bd6a-a7f0-49bb-844f-57b777bde2ed\") "
Oct 30 05:38:58.597700 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:58.597543    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47ff272a-ac5c-4f59-bf7b-5c0692d09386-dshm\" (UniqueName: \"kubernetes.io/empty-dir/47ff272a-ac5c-4f59-bf7b-5c0692d09386-dshm\") pod \"47ff272a-ac5c-4f59-bf7b-5c0692d09386\" (UID: \"47ff272a-ac5c-4f59-bf7b-5c0692d09386\") "
Oct 30 05:38:58.597700 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:58.597560    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47ff272a-ac5c-4f59-bf7b-5c0692d09386-home" pod "47ff272a-ac5c-4f59-bf7b-5c0692d09386" (UID: "47ff272a-ac5c-4f59-bf7b-5c0692d09386"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:38:58.597700 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:58.597574    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47ff272a-ac5c-4f59-bf7b-5c0692d09386-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47ff272a-ac5c-4f59-bf7b-5c0692d09386-tmp-dir\") pod \"47ff272a-ac5c-4f59-bf7b-5c0692d09386\" (UID: \"47ff272a-ac5c-4f59-bf7b-5c0692d09386\") "
Oct 30 05:38:58.597700 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:58.597612    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47ff272a-ac5c-4f59-bf7b-5c0692d09386-kube-api-access-bvgtk\" (UniqueName: \"kubernetes.io/projected/47ff272a-ac5c-4f59-bf7b-5c0692d09386-kube-api-access-bvgtk\") pod \"47ff272a-ac5c-4f59-bf7b-5c0692d09386\" (UID: \"47ff272a-ac5c-4f59-bf7b-5c0692d09386\") "
Oct 30 05:38:58.597700 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:58.597664    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f16bd6a-a7f0-49bb-844f-57b777bde2ed-home\" (UniqueName: \"kubernetes.io/empty-dir/6f16bd6a-a7f0-49bb-844f-57b777bde2ed-home\") pod \"6f16bd6a-a7f0-49bb-844f-57b777bde2ed\" (UID: \"6f16bd6a-a7f0-49bb-844f-57b777bde2ed\") "
Oct 30 05:38:58.598287 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:58.597714    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05eef4eb-f379-4dc7-a947-123bd85a4968-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05eef4eb-f379-4dc7-a947-123bd85a4968-tmp-dir\") pod \"05eef4eb-f379-4dc7-a947-123bd85a4968\" (UID: \"05eef4eb-f379-4dc7-a947-123bd85a4968\") "
Oct 30 05:38:58.598287 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:58.597743    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47ff272a-ac5c-4f59-bf7b-5c0692d09386-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47ff272a-ac5c-4f59-bf7b-5c0692d09386-model-cache\") pod \"47ff272a-ac5c-4f59-bf7b-5c0692d09386\" (UID: \"47ff272a-ac5c-4f59-bf7b-5c0692d09386\") "
Oct 30 05:38:58.598287 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:58.597754    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f16bd6a-a7f0-49bb-844f-57b777bde2ed-model-cache" pod "6f16bd6a-a7f0-49bb-844f-57b777bde2ed" (UID: "6f16bd6a-a7f0-49bb-844f-57b777bde2ed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:38:58.598287 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:58.597771    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05eef4eb-f379-4dc7-a947-123bd85a4968-dshm\" (UniqueName: \"kubernetes.io/empty-dir/05eef4eb-f379-4dc7-a947-123bd85a4968-dshm\") pod \"05eef4eb-f379-4dc7-a947-123bd85a4968\" (UID: \"05eef4eb-f379-4dc7-a947-123bd85a4968\") "
Oct 30 05:38:58.598287 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:58.598027    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05eef4eb-f379-4dc7-a947-123bd85a4968-home" pod "05eef4eb-f379-4dc7-a947-123bd85a4968" (UID: "05eef4eb-f379-4dc7-a947-123bd85a4968"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:38:58.598571 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:58.598333    5898 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f16bd6a-a7f0-49bb-844f-57b777bde2ed-model-cache\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:38:58.598571 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:58.598355    5898 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05eef4eb-f379-4dc7-a947-123bd85a4968-model-cache\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:38:58.598571 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:58.598369    5898 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05eef4eb-f379-4dc7-a947-123bd85a4968-home\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:38:58.598571 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:58.598382    5898 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47ff272a-ac5c-4f59-bf7b-5c0692d09386-home\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:38:58.599158 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:58.599131    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05eef4eb-f379-4dc7-a947-123bd85a4968-tmp-dir" pod "05eef4eb-f379-4dc7-a947-123bd85a4968" (UID: "05eef4eb-f379-4dc7-a947-123bd85a4968"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:38:58.599457 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:58.599433    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47ff272a-ac5c-4f59-bf7b-5c0692d09386-tmp-dir" pod "47ff272a-ac5c-4f59-bf7b-5c0692d09386" (UID: "47ff272a-ac5c-4f59-bf7b-5c0692d09386"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:38:58.599616 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:58.599585    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f16bd6a-a7f0-49bb-844f-57b777bde2ed-home" pod "6f16bd6a-a7f0-49bb-844f-57b777bde2ed" (UID: "6f16bd6a-a7f0-49bb-844f-57b777bde2ed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:38:58.599774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:58.599703    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47ff272a-ac5c-4f59-bf7b-5c0692d09386-model-cache" pod "47ff272a-ac5c-4f59-bf7b-5c0692d09386" (UID: "47ff272a-ac5c-4f59-bf7b-5c0692d09386"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:38:58.599774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:58.599748    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f16bd6a-a7f0-49bb-844f-57b777bde2ed-tmp-dir" pod "6f16bd6a-a7f0-49bb-844f-57b777bde2ed" (UID: "6f16bd6a-a7f0-49bb-844f-57b777bde2ed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:38:58.600681 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:58.600645    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f16bd6a-a7f0-49bb-844f-57b777bde2ed-dshm" pod "6f16bd6a-a7f0-49bb-844f-57b777bde2ed" (UID: "6f16bd6a-a7f0-49bb-844f-57b777bde2ed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:38:58.602591 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:58.602564    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05eef4eb-f379-4dc7-a947-123bd85a4968-dshm" pod "05eef4eb-f379-4dc7-a947-123bd85a4968" (UID: "05eef4eb-f379-4dc7-a947-123bd85a4968"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:38:58.603035 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:58.603001    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47ff272a-ac5c-4f59-bf7b-5c0692d09386-dshm" pod "47ff272a-ac5c-4f59-bf7b-5c0692d09386" (UID: "47ff272a-ac5c-4f59-bf7b-5c0692d09386"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:38:58.603241 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:58.603197    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f16bd6a-a7f0-49bb-844f-57b777bde2ed-tls-certs" pod "6f16bd6a-a7f0-49bb-844f-57b777bde2ed" (UID: "6f16bd6a-a7f0-49bb-844f-57b777bde2ed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:38:58.603241 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:58.603216    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05eef4eb-f379-4dc7-a947-123bd85a4968-tls-certs" pod "05eef4eb-f379-4dc7-a947-123bd85a4968" (UID: "05eef4eb-f379-4dc7-a947-123bd85a4968"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:38:58.603545 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:58.603513    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05eef4eb-f379-4dc7-a947-123bd85a4968-kube-api-access-rtpk5" pod "05eef4eb-f379-4dc7-a947-123bd85a4968" (UID: "05eef4eb-f379-4dc7-a947-123bd85a4968"). InnerVolumeSpecName "kube-api-access-rtpk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:38:58.603764 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:58.603736    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47ff272a-ac5c-4f59-bf7b-5c0692d09386-tls-certs" pod "47ff272a-ac5c-4f59-bf7b-5c0692d09386" (UID: "47ff272a-ac5c-4f59-bf7b-5c0692d09386"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:38:58.603866 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:58.603785    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47ff272a-ac5c-4f59-bf7b-5c0692d09386-kube-api-access-bvgtk" pod "47ff272a-ac5c-4f59-bf7b-5c0692d09386" (UID: "47ff272a-ac5c-4f59-bf7b-5c0692d09386"). InnerVolumeSpecName "kube-api-access-bvgtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:38:58.603866 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:58.603817    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f16bd6a-a7f0-49bb-844f-57b777bde2ed-kube-api-access-2j75z" pod "6f16bd6a-a7f0-49bb-844f-57b777bde2ed" (UID: "6f16bd6a-a7f0-49bb-844f-57b777bde2ed"). InnerVolumeSpecName "kube-api-access-2j75z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:38:58.699821 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:58.699728    5898 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47ff272a-ac5c-4f59-bf7b-5c0692d09386-tls-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:38:58.699821 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:58.699763    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtpk5\" (UniqueName: \"kubernetes.io/projected/05eef4eb-f379-4dc7-a947-123bd85a4968-kube-api-access-rtpk5\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:38:58.699821 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:58.699776    5898 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05eef4eb-f379-4dc7-a947-123bd85a4968-tls-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:38:58.699821 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:58.699789    5898 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f16bd6a-a7f0-49bb-844f-57b777bde2ed-dshm\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:38:58.699821 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:58.699800    5898 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f16bd6a-a7f0-49bb-844f-57b777bde2ed-tmp-dir\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:38:58.699821 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:58.699811    5898 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47ff272a-ac5c-4f59-bf7b-5c0692d09386-dshm\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:38:58.699821 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:58.699822    5898 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47ff272a-ac5c-4f59-bf7b-5c0692d09386-tmp-dir\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:38:58.700202 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:58.699835    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvgtk\" (UniqueName: \"kubernetes.io/projected/47ff272a-ac5c-4f59-bf7b-5c0692d09386-kube-api-access-bvgtk\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:38:58.700202 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:58.699846    5898 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f16bd6a-a7f0-49bb-844f-57b777bde2ed-home\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:38:58.700202 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:58.699860    5898 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05eef4eb-f379-4dc7-a947-123bd85a4968-tmp-dir\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:38:58.700202 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:58.699871    5898 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47ff272a-ac5c-4f59-bf7b-5c0692d09386-model-cache\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:38:58.700202 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:58.699882    5898 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05eef4eb-f379-4dc7-a947-123bd85a4968-dshm\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:38:58.700202 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:58.699892    5898 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f16bd6a-a7f0-49bb-844f-57b777bde2ed-tls-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:38:58.700202 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:58.699918    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2j75z\" (UniqueName: \"kubernetes.io/projected/6f16bd6a-a7f0-49bb-844f-57b777bde2ed-kube-api-access-2j75z\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:38:58.827127 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:58.827093    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:38:59.130422 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:59.130387    5898 generic.go:356] "Generic (PLEG): container finished" podID="47ff272a-ac5c-4f59-bf7b-5c0692d09386" containerID="5ea897a4d77a647b44003d22b196cc3136b848404e7f753cb8fd5d8a9a4bd7c8" exitCode=0
Oct 30 05:38:59.130592 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:59.130432    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6"
Oct 30 05:38:59.130592 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:59.130468    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6" event={"ID":"47ff272a-ac5c-4f59-bf7b-5c0692d09386","Type":"ContainerDied","Data":"5ea897a4d77a647b44003d22b196cc3136b848404e7f753cb8fd5d8a9a4bd7c8"}
Oct 30 05:38:59.130592 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:59.130506    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6" event={"ID":"47ff272a-ac5c-4f59-bf7b-5c0692d09386","Type":"ContainerDied","Data":"24e7dced4f47653ec987bd2fe8dd3c5c5149650c88af1ce06d5a282a295efe81"}
Oct 30 05:38:59.130592 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:59.130525    5898 scope.go:122] "RemoveContainer" containerID="5ea897a4d77a647b44003d22b196cc3136b848404e7f753cb8fd5d8a9a4bd7c8"
Oct 30 05:38:59.131918 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:59.131873    5898 generic.go:356] "Generic (PLEG): container finished" podID="6f16bd6a-a7f0-49bb-844f-57b777bde2ed" containerID="9e43b28fab1725efc5f8953ba3c552342e7224362b718c643f783f26beb042b2" exitCode=0
Oct 30 05:38:59.132002 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:59.131954    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb"
Oct 30 05:38:59.132067 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:59.132026    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb" event={"ID":"6f16bd6a-a7f0-49bb-844f-57b777bde2ed","Type":"ContainerDied","Data":"9e43b28fab1725efc5f8953ba3c552342e7224362b718c643f783f26beb042b2"}
Oct 30 05:38:59.132067 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:59.132051    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb" event={"ID":"6f16bd6a-a7f0-49bb-844f-57b777bde2ed","Type":"ContainerDied","Data":"cdf000ba236a68d99a0632777c9d516f67bc0982c56d66a6cb8467c0e0134188"}
Oct 30 05:38:59.133651 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:59.133628    5898 generic.go:356] "Generic (PLEG): container finished" podID="05eef4eb-f379-4dc7-a947-123bd85a4968" containerID="4f15f34f28e86ff388667b9f6ae2d0c86f1c6931168dfb4f72443c054797aef8" exitCode=0
Oct 30 05:38:59.133770 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:59.133657    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49" event={"ID":"05eef4eb-f379-4dc7-a947-123bd85a4968","Type":"ContainerDied","Data":"4f15f34f28e86ff388667b9f6ae2d0c86f1c6931168dfb4f72443c054797aef8"}
Oct 30 05:38:59.133770 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:59.133688    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49" event={"ID":"05eef4eb-f379-4dc7-a947-123bd85a4968","Type":"ContainerDied","Data":"b9d069cd11fe41b79c6ed98ce4ad656ef542f11338edd54ac01c42e5f06d177a"}
Oct 30 05:38:59.133770 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:59.133667    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49"
Oct 30 05:38:59.140765 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:59.140748    5898 scope.go:122] "RemoveContainer" containerID="5ea897a4d77a647b44003d22b196cc3136b848404e7f753cb8fd5d8a9a4bd7c8"
Oct 30 05:38:59.141035 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:38:59.141014    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ea897a4d77a647b44003d22b196cc3136b848404e7f753cb8fd5d8a9a4bd7c8\": container with ID starting with 5ea897a4d77a647b44003d22b196cc3136b848404e7f753cb8fd5d8a9a4bd7c8 not found: ID does not exist" containerID="5ea897a4d77a647b44003d22b196cc3136b848404e7f753cb8fd5d8a9a4bd7c8"
Oct 30 05:38:59.141111 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:59.141043    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ea897a4d77a647b44003d22b196cc3136b848404e7f753cb8fd5d8a9a4bd7c8"} err="failed to get container status \"5ea897a4d77a647b44003d22b196cc3136b848404e7f753cb8fd5d8a9a4bd7c8\": rpc error: code = NotFound desc = could not find container \"5ea897a4d77a647b44003d22b196cc3136b848404e7f753cb8fd5d8a9a4bd7c8\": container with ID starting with 5ea897a4d77a647b44003d22b196cc3136b848404e7f753cb8fd5d8a9a4bd7c8 not found: ID does not exist"
Oct 30 05:38:59.141111 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:59.141059    5898 scope.go:122] "RemoveContainer" containerID="9e43b28fab1725efc5f8953ba3c552342e7224362b718c643f783f26beb042b2"
Oct 30 05:38:59.149267 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:59.149250    5898 scope.go:122] "RemoveContainer" containerID="9e43b28fab1725efc5f8953ba3c552342e7224362b718c643f783f26beb042b2"
Oct 30 05:38:59.149527 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:38:59.149507    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e43b28fab1725efc5f8953ba3c552342e7224362b718c643f783f26beb042b2\": container with ID starting with 9e43b28fab1725efc5f8953ba3c552342e7224362b718c643f783f26beb042b2 not found: ID does not exist" containerID="9e43b28fab1725efc5f8953ba3c552342e7224362b718c643f783f26beb042b2"
Oct 30 05:38:59.149571 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:59.149537    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e43b28fab1725efc5f8953ba3c552342e7224362b718c643f783f26beb042b2"} err="failed to get container status \"9e43b28fab1725efc5f8953ba3c552342e7224362b718c643f783f26beb042b2\": rpc error: code = NotFound desc = could not find container \"9e43b28fab1725efc5f8953ba3c552342e7224362b718c643f783f26beb042b2\": container with ID starting with 9e43b28fab1725efc5f8953ba3c552342e7224362b718c643f783f26beb042b2 not found: ID does not exist"
Oct 30 05:38:59.149571 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:59.149553    5898 scope.go:122] "RemoveContainer" containerID="4f15f34f28e86ff388667b9f6ae2d0c86f1c6931168dfb4f72443c054797aef8"
Oct 30 05:38:59.157292 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:59.157272    5898 scope.go:122] "RemoveContainer" containerID="4f15f34f28e86ff388667b9f6ae2d0c86f1c6931168dfb4f72443c054797aef8"
Oct 30 05:38:59.157593 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:38:59.157569    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f15f34f28e86ff388667b9f6ae2d0c86f1c6931168dfb4f72443c054797aef8\": container with ID starting with 4f15f34f28e86ff388667b9f6ae2d0c86f1c6931168dfb4f72443c054797aef8 not found: ID does not exist" containerID="4f15f34f28e86ff388667b9f6ae2d0c86f1c6931168dfb4f72443c054797aef8"
Oct 30 05:38:59.157659 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:59.157604    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f15f34f28e86ff388667b9f6ae2d0c86f1c6931168dfb4f72443c054797aef8"} err="failed to get container status \"4f15f34f28e86ff388667b9f6ae2d0c86f1c6931168dfb4f72443c054797aef8\": rpc error: code = NotFound desc = could not find container \"4f15f34f28e86ff388667b9f6ae2d0c86f1c6931168dfb4f72443c054797aef8\": container with ID starting with 4f15f34f28e86ff388667b9f6ae2d0c86f1c6931168dfb4f72443c054797aef8 not found: ID does not exist"
Oct 30 05:38:59.181705 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:59.181682    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb"]
Oct 30 05:38:59.195066 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:59.195043    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-397179/premium-w2-397179-kserve-797dbcc7bd-d9jwb"]
Oct 30 05:38:59.237071 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:59.237047    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49"]
Oct 30 05:38:59.242710 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:59.242685    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-397179/simulator-w2-397179-kserve-57b67456f7-n4g49"]
Oct 30 05:38:59.289558 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:59.289533    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6"]
Oct 30 05:38:59.310230 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:38:59.310205    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-397179/distinct-w2-397179-kserve-56547fb455-flrg6"]
Oct 30 05:39:00.843480 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:00.843447    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:00.964999 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:00.964961    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05eef4eb-f379-4dc7-a947-123bd85a4968" path="/var/lib/kubelet/pods/05eef4eb-f379-4dc7-a947-123bd85a4968/volumes"
Oct 30 05:39:00.965855 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:00.965834    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47ff272a-ac5c-4f59-bf7b-5c0692d09386" path="/var/lib/kubelet/pods/47ff272a-ac5c-4f59-bf7b-5c0692d09386/volumes"
Oct 30 05:39:00.966495 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:00.966475    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f16bd6a-a7f0-49bb-844f-57b777bde2ed" path="/var/lib/kubelet/pods/6f16bd6a-a7f0-49bb-844f-57b777bde2ed/volumes"
Oct 30 05:39:00.966821 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:00.966798    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:39:01.740479 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:01.740440    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:01.940423 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:01.940389    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:02.874162 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:02.874133    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:04.085882 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:04.085848    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:04.205154 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:04.205127    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:04.413707 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:04.413627    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4"]
Oct 30 05:39:04.413934 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:04.413885    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4" podUID="edceaaa4-f231-474e-85e1-b877ede202ef" containerName="istio-proxy" containerID="cri-o://dea7f224f5dae13e8cdb887c420e975d8d10123af82b3bd4fe012a4926eca760" gracePeriod=30
Oct 30 05:39:04.444514 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:04.444478    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:04.698371 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:04.698280    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:06.693789 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:06.693735    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:08.273067 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:08.273038    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:08.376501 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:08.376457    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:08.605951 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:08.605883    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:09.274016 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:09.273981    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:09.687079 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:09.687051    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4"
Oct 30 05:39:09.739247 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:09.739214    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edceaaa4-f231-474e-85e1-b877ede202ef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edceaaa4-f231-474e-85e1-b877ede202ef-workload-certs\") pod \"edceaaa4-f231-474e-85e1-b877ede202ef\" (UID: \"edceaaa4-f231-474e-85e1-b877ede202ef\") "
Oct 30 05:39:09.739425 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:09.739291    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edceaaa4-f231-474e-85e1-b877ede202ef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edceaaa4-f231-474e-85e1-b877ede202ef-istio-data\") pod \"edceaaa4-f231-474e-85e1-b877ede202ef\" (UID: \"edceaaa4-f231-474e-85e1-b877ede202ef\") "
Oct 30 05:39:09.739425 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:09.739340    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edceaaa4-f231-474e-85e1-b877ede202ef-istio-token\" (UniqueName: \"kubernetes.io/projected/edceaaa4-f231-474e-85e1-b877ede202ef-istio-token\") pod \"edceaaa4-f231-474e-85e1-b877ede202ef\" (UID: \"edceaaa4-f231-474e-85e1-b877ede202ef\") "
Oct 30 05:39:09.739425 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:09.739413    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edceaaa4-f231-474e-85e1-b877ede202ef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edceaaa4-f231-474e-85e1-b877ede202ef-workload-socket\") pod \"edceaaa4-f231-474e-85e1-b877ede202ef\" (UID: \"edceaaa4-f231-474e-85e1-b877ede202ef\") "
Oct 30 05:39:09.739631 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:09.739523    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edceaaa4-f231-474e-85e1-b877ede202ef-workload-certs" pod "edceaaa4-f231-474e-85e1-b877ede202ef" (UID: "edceaaa4-f231-474e-85e1-b877ede202ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:39:09.739631 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:09.739556    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edceaaa4-f231-474e-85e1-b877ede202ef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edceaaa4-f231-474e-85e1-b877ede202ef-istiod-ca-cert\") pod \"edceaaa4-f231-474e-85e1-b877ede202ef\" (UID: \"edceaaa4-f231-474e-85e1-b877ede202ef\") "
Oct 30 05:39:09.739631 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:09.739586    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edceaaa4-f231-474e-85e1-b877ede202ef-istio-data" pod "edceaaa4-f231-474e-85e1-b877ede202ef" (UID: "edceaaa4-f231-474e-85e1-b877ede202ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:39:09.739631 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:09.739623    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edceaaa4-f231-474e-85e1-b877ede202ef-kube-api-access-8p74m\" (UniqueName: \"kubernetes.io/projected/edceaaa4-f231-474e-85e1-b877ede202ef-kube-api-access-8p74m\") pod \"edceaaa4-f231-474e-85e1-b877ede202ef\" (UID: \"edceaaa4-f231-474e-85e1-b877ede202ef\") "
Oct 30 05:39:09.739835 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:09.739659    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edceaaa4-f231-474e-85e1-b877ede202ef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edceaaa4-f231-474e-85e1-b877ede202ef-istio-envoy\") pod \"edceaaa4-f231-474e-85e1-b877ede202ef\" (UID: \"edceaaa4-f231-474e-85e1-b877ede202ef\") "
Oct 30 05:39:09.739835 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:09.739695    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edceaaa4-f231-474e-85e1-b877ede202ef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edceaaa4-f231-474e-85e1-b877ede202ef-credential-socket\") pod \"edceaaa4-f231-474e-85e1-b877ede202ef\" (UID: \"edceaaa4-f231-474e-85e1-b877ede202ef\") "
Oct 30 05:39:09.739835 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:09.739731    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edceaaa4-f231-474e-85e1-b877ede202ef-workload-socket" pod "edceaaa4-f231-474e-85e1-b877ede202ef" (UID: "edceaaa4-f231-474e-85e1-b877ede202ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:39:09.739835 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:09.739765    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/edceaaa4-f231-474e-85e1-b877ede202ef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edceaaa4-f231-474e-85e1-b877ede202ef-istio-podinfo\") pod \"edceaaa4-f231-474e-85e1-b877ede202ef\" (UID: \"edceaaa4-f231-474e-85e1-b877ede202ef\") "
Oct 30 05:39:09.740085 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:09.739955    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edceaaa4-f231-474e-85e1-b877ede202ef-credential-socket" pod "edceaaa4-f231-474e-85e1-b877ede202ef" (UID: "edceaaa4-f231-474e-85e1-b877ede202ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:39:09.740085 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:09.739980    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edceaaa4-f231-474e-85e1-b877ede202ef-istiod-ca-cert" pod "edceaaa4-f231-474e-85e1-b877ede202ef" (UID: "edceaaa4-f231-474e-85e1-b877ede202ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:39:09.740282 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:09.740252    5898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edceaaa4-f231-474e-85e1-b877ede202ef-workload-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:09.740432 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:09.740287    5898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edceaaa4-f231-474e-85e1-b877ede202ef-istiod-ca-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:09.740432 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:09.740302    5898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edceaaa4-f231-474e-85e1-b877ede202ef-credential-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:09.740432 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:09.740317    5898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edceaaa4-f231-474e-85e1-b877ede202ef-workload-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:09.740432 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:09.740330    5898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edceaaa4-f231-474e-85e1-b877ede202ef-istio-data\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:09.741960 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:09.741887    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edceaaa4-f231-474e-85e1-b877ede202ef-istio-token" pod "edceaaa4-f231-474e-85e1-b877ede202ef" (UID: "edceaaa4-f231-474e-85e1-b877ede202ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:39:09.741960 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:09.741937    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edceaaa4-f231-474e-85e1-b877ede202ef-kube-api-access-8p74m" pod "edceaaa4-f231-474e-85e1-b877ede202ef" (UID: "edceaaa4-f231-474e-85e1-b877ede202ef"). InnerVolumeSpecName "kube-api-access-8p74m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:39:09.742101 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:09.741971    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/edceaaa4-f231-474e-85e1-b877ede202ef-istio-podinfo" pod "edceaaa4-f231-474e-85e1-b877ede202ef" (UID: "edceaaa4-f231-474e-85e1-b877ede202ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:39:09.742196 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:09.742169    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edceaaa4-f231-474e-85e1-b877ede202ef-istio-envoy" pod "edceaaa4-f231-474e-85e1-b877ede202ef" (UID: "edceaaa4-f231-474e-85e1-b877ede202ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:39:09.841965 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:09.841929    5898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edceaaa4-f231-474e-85e1-b877ede202ef-istio-podinfo\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:09.841965 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:09.841958    5898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edceaaa4-f231-474e-85e1-b877ede202ef-istio-token\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:09.841965 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:09.841967    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p74m\" (UniqueName: \"kubernetes.io/projected/edceaaa4-f231-474e-85e1-b877ede202ef-kube-api-access-8p74m\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:09.842209 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:09.841977    5898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edceaaa4-f231-474e-85e1-b877ede202ef-istio-envoy\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:10.228958 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:10.228823    5898 generic.go:356] "Generic (PLEG): container finished" podID="edceaaa4-f231-474e-85e1-b877ede202ef" containerID="dea7f224f5dae13e8cdb887c420e975d8d10123af82b3bd4fe012a4926eca760" exitCode=0
Oct 30 05:39:10.228958 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:10.228882    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4"
Oct 30 05:39:10.228958 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:10.228928    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4" event={"ID":"edceaaa4-f231-474e-85e1-b877ede202ef","Type":"ContainerDied","Data":"dea7f224f5dae13e8cdb887c420e975d8d10123af82b3bd4fe012a4926eca760"}
Oct 30 05:39:10.229335 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:10.228986    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4" event={"ID":"edceaaa4-f231-474e-85e1-b877ede202ef","Type":"ContainerDied","Data":"a1ceeca5b289c5d539f7067f4767608761fc50a51be1ef48bbf8a1142742a538"}
Oct 30 05:39:10.229335 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:10.229016    5898 scope.go:122] "RemoveContainer" containerID="dea7f224f5dae13e8cdb887c420e975d8d10123af82b3bd4fe012a4926eca760"
Oct 30 05:39:10.249339 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:10.249315    5898 scope.go:122] "RemoveContainer" containerID="dea7f224f5dae13e8cdb887c420e975d8d10123af82b3bd4fe012a4926eca760"
Oct 30 05:39:10.252459 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:39:10.252425    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dea7f224f5dae13e8cdb887c420e975d8d10123af82b3bd4fe012a4926eca760\": container with ID starting with dea7f224f5dae13e8cdb887c420e975d8d10123af82b3bd4fe012a4926eca760 not found: ID does not exist" containerID="dea7f224f5dae13e8cdb887c420e975d8d10123af82b3bd4fe012a4926eca760"
Oct 30 05:39:10.252599 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:10.252469    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dea7f224f5dae13e8cdb887c420e975d8d10123af82b3bd4fe012a4926eca760"} err="failed to get container status \"dea7f224f5dae13e8cdb887c420e975d8d10123af82b3bd4fe012a4926eca760\": rpc error: code = NotFound desc = could not find container \"dea7f224f5dae13e8cdb887c420e975d8d10123af82b3bd4fe012a4926eca760\": container with ID starting with dea7f224f5dae13e8cdb887c420e975d8d10123af82b3bd4fe012a4926eca760 not found: ID does not exist"
Oct 30 05:39:10.260173 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:10.260132    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:10.286646 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:10.286608    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4"]
Oct 30 05:39:10.296418 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:10.296387    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b21507-openshift-default-7dcf4797cf-4vgk4"]
Oct 30 05:39:10.387325 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:10.387290    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:10.492810 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:10.492726    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:10.649716 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:10.649682    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:10.790585 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:10.790491    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:10.889885 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:10.889853    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edceaaa4-f231-474e-85e1-b877ede202ef" path="/var/lib/kubelet/pods/edceaaa4-f231-474e-85e1-b877ede202ef/volumes"
Oct 30 05:39:10.905519 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:10.905473    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:11.026115 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:11.026082    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34722e-554d6dcb9f-rxxd9"]
Oct 30 05:39:11.026355 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:11.026328    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34722e-554d6dcb9f-rxxd9" podUID="8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff" containerName="maas-api" containerID="cri-o://5a11dc9dfdf7726122edca5ca0898744893254a179be1830a4bef948d731b0a4" gracePeriod=30
Oct 30 05:39:11.119948 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:11.115626    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:11.239867 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:11.239818    5898 generic.go:356] "Generic (PLEG): container finished" podID="8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff" containerID="5a11dc9dfdf7726122edca5ca0898744893254a179be1830a4bef948d731b0a4" exitCode=0
Oct 30 05:39:11.240074 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:11.240000    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34722e-554d6dcb9f-rxxd9" event={"ID":"8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff","Type":"ContainerDied","Data":"5a11dc9dfdf7726122edca5ca0898744893254a179be1830a4bef948d731b0a4"}
Oct 30 05:39:11.246404 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:11.246355    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:11.347444 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:11.346418    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34722e-554d6dcb9f-rxxd9"
Oct 30 05:39:11.379625 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:11.378438    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:11.477621 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:11.477584    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff-maas-api-tls\") pod \"8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff\" (UID: \"8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff\") "
Oct 30 05:39:11.477803 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:11.477626    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff-kube-api-access-994nm\" (UniqueName: \"kubernetes.io/projected/8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff-kube-api-access-994nm\") pod \"8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff\" (UID: \"8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff\") "
Oct 30 05:39:11.477803 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:11.477659    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff-metrics-certs\") pod \"8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff\" (UID: \"8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff\") "
Oct 30 05:39:11.477803 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:11.477758    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff-openshift-service-ca\") pod \"8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff\" (UID: \"8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff\") "
Oct 30 05:39:11.478183 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:11.478155    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff-openshift-service-ca" pod "8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff" (UID: "8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:39:11.479799 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:11.479775    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff-metrics-certs" pod "8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff" (UID: "8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:39:11.480234 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:11.480211    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff-maas-api-tls" pod "8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff" (UID: "8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:39:11.480234 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:11.480224    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff-kube-api-access-994nm" pod "8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff" (UID: "8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff"). InnerVolumeSpecName "kube-api-access-994nm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:39:11.488994 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:11.488951    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:11.582374 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:11.580968    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff-maas-api-tls\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:11.582374 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:11.581003    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-994nm\" (UniqueName: \"kubernetes.io/projected/8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff-kube-api-access-994nm\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:11.582374 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:11.581019    5898 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff-metrics-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:11.582374 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:11.581034    5898 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff-openshift-service-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:11.747074 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:11.746993    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:11.846403 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:11.846375    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:11.959924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:11.959861    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:12.246804 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:12.246777    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34722e-554d6dcb9f-rxxd9"
Oct 30 05:39:12.247010 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:12.246781    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34722e-554d6dcb9f-rxxd9" event={"ID":"8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff","Type":"ContainerDied","Data":"25440a5e63171687771c23c10e2a1989b1d264b217d8b343b6a8235eddfa04db"}
Oct 30 05:39:12.247010 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:12.246923    5898 scope.go:122] "RemoveContainer" containerID="5a11dc9dfdf7726122edca5ca0898744893254a179be1830a4bef948d731b0a4"
Oct 30 05:39:12.298127 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:12.298093    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34722e-554d6dcb9f-rxxd9"]
Oct 30 05:39:12.309052 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:12.309028    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34722e-554d6dcb9f-rxxd9"]
Oct 30 05:39:12.482591 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:12.482558    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:12.889790 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:12.889758    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff" path="/var/lib/kubelet/pods/8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff/volumes"
Oct 30 05:39:13.765991 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:13.765957    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:13.926468 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:13.926434    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:39:14.167183 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:14.167152    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb"]
Oct 30 05:39:14.167548 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:14.167395    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb" podUID="8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f" containerName="istio-proxy" containerID="cri-o://f41fe135137090ccc037d32facef08d38129a7fcef77289a60839c9df7d540fb" gracePeriod=30
Oct 30 05:39:14.178435 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:14.178401    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh"]
Oct 30 05:39:14.178669 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:14.178647    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh" podUID="b4dc6435-93bb-4b45-bdd5-3a80868e15e3" containerName="istio-proxy" containerID="cri-o://98c683bf5f41dcf1b722bc2d41507fa4f24694bebd68dcfa7c7957a9a7891dfd" gracePeriod=30
Oct 30 05:39:15.757422 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:15.757382    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:16.259340 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:16.259307    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:16.365539 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:16.365508    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:17.599764 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:17.599729    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:18.370489 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:18.370455    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:18.513937 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:18.512686    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:18.757262 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:18.757176    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:19.277534 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:19.277502    5898 patch_prober.go:29] interesting pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 30 05:39:19.277534 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:19.277522    5898 patch_prober.go:29] interesting pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 30 05:39:19.277713 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:19.277586    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb" podUID="8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 30 05:39:19.277713 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:19.277595    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh" podUID="b4dc6435-93bb-4b45-bdd5-3a80868e15e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 30 05:39:19.294834 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:19.294809    5898 generic.go:356] "Generic (PLEG): container finished" podID="8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f" containerID="f41fe135137090ccc037d32facef08d38129a7fcef77289a60839c9df7d540fb" exitCode=0
Oct 30 05:39:19.294967 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:19.294852    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb" event={"ID":"8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f","Type":"ContainerDied","Data":"f41fe135137090ccc037d32facef08d38129a7fcef77289a60839c9df7d540fb"}
Oct 30 05:39:19.458193 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:19.458165    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb"
Oct 30 05:39:19.461408 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:19.461386    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh"
Oct 30 05:39:19.591823 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:19.591790    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f-istio-podinfo\") pod \"8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f\" (UID: \"8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f\") "
Oct 30 05:39:19.592351 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:19.591854    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f-kube-api-access-m9f86\" (UniqueName: \"kubernetes.io/projected/8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f-kube-api-access-m9f86\") pod \"8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f\" (UID: \"8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f\") "
Oct 30 05:39:19.592351 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:19.592283    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f-istio-data\") pod \"8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f\" (UID: \"8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f\") "
Oct 30 05:39:19.592511 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:19.592356    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f-workload-certs\") pod \"8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f\" (UID: \"8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f\") "
Oct 30 05:39:19.592511 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:19.592401    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4dc6435-93bb-4b45-bdd5-3a80868e15e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4dc6435-93bb-4b45-bdd5-3a80868e15e3-istio-data\") pod \"b4dc6435-93bb-4b45-bdd5-3a80868e15e3\" (UID: \"b4dc6435-93bb-4b45-bdd5-3a80868e15e3\") "
Oct 30 05:39:19.592511 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:19.592438    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4dc6435-93bb-4b45-bdd5-3a80868e15e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4dc6435-93bb-4b45-bdd5-3a80868e15e3-credential-socket\") pod \"b4dc6435-93bb-4b45-bdd5-3a80868e15e3\" (UID: \"b4dc6435-93bb-4b45-bdd5-3a80868e15e3\") "
Oct 30 05:39:19.592511 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:19.592471    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b4dc6435-93bb-4b45-bdd5-3a80868e15e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4dc6435-93bb-4b45-bdd5-3a80868e15e3-istio-podinfo\") pod \"b4dc6435-93bb-4b45-bdd5-3a80868e15e3\" (UID: \"b4dc6435-93bb-4b45-bdd5-3a80868e15e3\") "
Oct 30 05:39:19.592511 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:19.592502    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f-istiod-ca-cert\") pod \"8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f\" (UID: \"8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f\") "
Oct 30 05:39:19.592778 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:19.592554    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f-credential-socket\") pod \"8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f\" (UID: \"8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f\") "
Oct 30 05:39:19.592778 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:19.592611    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4dc6435-93bb-4b45-bdd5-3a80868e15e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4dc6435-93bb-4b45-bdd5-3a80868e15e3-istiod-ca-cert\") pod \"b4dc6435-93bb-4b45-bdd5-3a80868e15e3\" (UID: \"b4dc6435-93bb-4b45-bdd5-3a80868e15e3\") "
Oct 30 05:39:19.592778 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:19.592685    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f-istio-envoy\") pod \"8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f\" (UID: \"8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f\") "
Oct 30 05:39:19.592778 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:19.592713    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f-istio-token\" (UniqueName: \"kubernetes.io/projected/8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f-istio-token\") pod \"8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f\" (UID: \"8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f\") "
Oct 30 05:39:19.592778 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:19.592730    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4dc6435-93bb-4b45-bdd5-3a80868e15e3-istio-data" pod "b4dc6435-93bb-4b45-bdd5-3a80868e15e3" (UID: "b4dc6435-93bb-4b45-bdd5-3a80868e15e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:39:19.593286 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:19.592783    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4dc6435-93bb-4b45-bdd5-3a80868e15e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4dc6435-93bb-4b45-bdd5-3a80868e15e3-workload-certs\") pod \"b4dc6435-93bb-4b45-bdd5-3a80868e15e3\" (UID: \"b4dc6435-93bb-4b45-bdd5-3a80868e15e3\") "
Oct 30 05:39:19.593286 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:19.592837    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4dc6435-93bb-4b45-bdd5-3a80868e15e3-kube-api-access-hmxl4\" (UniqueName: \"kubernetes.io/projected/b4dc6435-93bb-4b45-bdd5-3a80868e15e3-kube-api-access-hmxl4\") pod \"b4dc6435-93bb-4b45-bdd5-3a80868e15e3\" (UID: \"b4dc6435-93bb-4b45-bdd5-3a80868e15e3\") "
Oct 30 05:39:19.593286 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:19.592870    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4dc6435-93bb-4b45-bdd5-3a80868e15e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4dc6435-93bb-4b45-bdd5-3a80868e15e3-workload-socket\") pod \"b4dc6435-93bb-4b45-bdd5-3a80868e15e3\" (UID: \"b4dc6435-93bb-4b45-bdd5-3a80868e15e3\") "
Oct 30 05:39:19.593286 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:19.592915    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4dc6435-93bb-4b45-bdd5-3a80868e15e3-istio-token\" (UniqueName: \"kubernetes.io/projected/b4dc6435-93bb-4b45-bdd5-3a80868e15e3-istio-token\") pod \"b4dc6435-93bb-4b45-bdd5-3a80868e15e3\" (UID: \"b4dc6435-93bb-4b45-bdd5-3a80868e15e3\") "
Oct 30 05:39:19.593286 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:19.592976    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f-workload-socket\") pod \"8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f\" (UID: \"8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f\") "
Oct 30 05:39:19.593286 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:19.593005    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4dc6435-93bb-4b45-bdd5-3a80868e15e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4dc6435-93bb-4b45-bdd5-3a80868e15e3-istio-envoy\") pod \"b4dc6435-93bb-4b45-bdd5-3a80868e15e3\" (UID: \"b4dc6435-93bb-4b45-bdd5-3a80868e15e3\") "
Oct 30 05:39:19.593286 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:19.593066    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f-istio-data" pod "8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f" (UID: "8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:39:19.593286 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:19.593228    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4dc6435-93bb-4b45-bdd5-3a80868e15e3-credential-socket" pod "b4dc6435-93bb-4b45-bdd5-3a80868e15e3" (UID: "b4dc6435-93bb-4b45-bdd5-3a80868e15e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:39:19.593286 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:19.593235    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f-workload-certs" pod "8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f" (UID: "8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:39:19.594286 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:19.593446    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f-credential-socket" pod "8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f" (UID: "8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:39:19.594286 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:19.593654    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4dc6435-93bb-4b45-bdd5-3a80868e15e3-workload-certs" pod "b4dc6435-93bb-4b45-bdd5-3a80868e15e3" (UID: "b4dc6435-93bb-4b45-bdd5-3a80868e15e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:39:19.594286 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:19.593735    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4dc6435-93bb-4b45-bdd5-3a80868e15e3-istiod-ca-cert" pod "b4dc6435-93bb-4b45-bdd5-3a80868e15e3" (UID: "b4dc6435-93bb-4b45-bdd5-3a80868e15e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:39:19.594286 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:19.594242    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f-istiod-ca-cert" pod "8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f" (UID: "8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:39:19.594687 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:19.594428    5898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f-workload-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:19.594687 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:19.594453    5898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4dc6435-93bb-4b45-bdd5-3a80868e15e3-istio-data\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:19.594687 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:19.594466    5898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4dc6435-93bb-4b45-bdd5-3a80868e15e3-credential-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:19.594687 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:19.594482    5898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f-istiod-ca-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:19.594687 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:19.594497    5898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f-credential-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:19.594687 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:19.594512    5898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4dc6435-93bb-4b45-bdd5-3a80868e15e3-istiod-ca-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:19.594687 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:19.594525    5898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4dc6435-93bb-4b45-bdd5-3a80868e15e3-workload-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:19.594687 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:19.594537    5898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f-istio-data\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:19.594687 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:19.594561    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f-workload-socket" pod "8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f" (UID: "8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:39:19.595203 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:19.594769    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4dc6435-93bb-4b45-bdd5-3a80868e15e3-workload-socket" pod "b4dc6435-93bb-4b45-bdd5-3a80868e15e3" (UID: "b4dc6435-93bb-4b45-bdd5-3a80868e15e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:39:19.595444 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:19.595389    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f-kube-api-access-m9f86" pod "8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f" (UID: "8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f"). InnerVolumeSpecName "kube-api-access-m9f86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:39:19.596438 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:19.596412    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4dc6435-93bb-4b45-bdd5-3a80868e15e3-kube-api-access-hmxl4" pod "b4dc6435-93bb-4b45-bdd5-3a80868e15e3" (UID: "b4dc6435-93bb-4b45-bdd5-3a80868e15e3"). InnerVolumeSpecName "kube-api-access-hmxl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:39:19.596536 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:19.596432    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f-istio-podinfo" pod "8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f" (UID: "8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:39:19.596639 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:19.596624    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4dc6435-93bb-4b45-bdd5-3a80868e15e3-istio-token" pod "b4dc6435-93bb-4b45-bdd5-3a80868e15e3" (UID: "b4dc6435-93bb-4b45-bdd5-3a80868e15e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:39:19.596694 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:19.596641    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4dc6435-93bb-4b45-bdd5-3a80868e15e3-istio-envoy" pod "b4dc6435-93bb-4b45-bdd5-3a80868e15e3" (UID: "b4dc6435-93bb-4b45-bdd5-3a80868e15e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:39:19.596942 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:19.596925    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f-istio-token" pod "8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f" (UID: "8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:39:19.597563 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:19.597549    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4dc6435-93bb-4b45-bdd5-3a80868e15e3-istio-podinfo" pod "b4dc6435-93bb-4b45-bdd5-3a80868e15e3" (UID: "b4dc6435-93bb-4b45-bdd5-3a80868e15e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:39:19.598198 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:19.598179    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f-istio-envoy" pod "8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f" (UID: "8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:39:19.680957 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:19.680928    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:19.695502 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:19.695437    5898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4dc6435-93bb-4b45-bdd5-3a80868e15e3-istio-token\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:19.695502 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:19.695466    5898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f-workload-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:19.695502 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:19.695482    5898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4dc6435-93bb-4b45-bdd5-3a80868e15e3-istio-envoy\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:19.695502 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:19.695496    5898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f-istio-podinfo\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:19.695790 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:19.695509    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9f86\" (UniqueName: \"kubernetes.io/projected/8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f-kube-api-access-m9f86\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:19.695790 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:19.695522    5898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4dc6435-93bb-4b45-bdd5-3a80868e15e3-istio-podinfo\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:19.695790 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:19.695534    5898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f-istio-envoy\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:19.695790 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:19.695546    5898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f-istio-token\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:19.695790 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:19.695560    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmxl4\" (UniqueName: \"kubernetes.io/projected/b4dc6435-93bb-4b45-bdd5-3a80868e15e3-kube-api-access-hmxl4\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:19.695790 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:19.695579    5898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4dc6435-93bb-4b45-bdd5-3a80868e15e3-workload-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:19.893029 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:19.892994    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:20.036561 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:20.036468    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:20.168499 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:20.168462    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:20.260039 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:20.260007    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp"]
Oct 30 05:39:20.260282 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:20.260250    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp" podUID="fa97913c-19e1-432c-9bd2-b1ce3278f7a5" containerName="istio-proxy" containerID="cri-o://0d431a2552cdab88b3e21fa66e0a7a7321fbc890b500e7a693d010be9e80209a" gracePeriod=30
Oct 30 05:39:20.295118 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:20.295028    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:20.301206 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:20.301182    5898 generic.go:356] "Generic (PLEG): container finished" podID="b4dc6435-93bb-4b45-bdd5-3a80868e15e3" containerID="98c683bf5f41dcf1b722bc2d41507fa4f24694bebd68dcfa7c7957a9a7891dfd" exitCode=0
Oct 30 05:39:20.301341 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:20.301235    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh"
Oct 30 05:39:20.301341 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:20.301273    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh" event={"ID":"b4dc6435-93bb-4b45-bdd5-3a80868e15e3","Type":"ContainerDied","Data":"98c683bf5f41dcf1b722bc2d41507fa4f24694bebd68dcfa7c7957a9a7891dfd"}
Oct 30 05:39:20.301341 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:20.301326    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh" event={"ID":"b4dc6435-93bb-4b45-bdd5-3a80868e15e3","Type":"ContainerDied","Data":"2c645020fdf4b79ffeaa80e9eb4a45f0b4d83795ef8bed32c42f8a4c487c6d3b"}
Oct 30 05:39:20.301518 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:20.301358    5898 scope.go:122] "RemoveContainer" containerID="98c683bf5f41dcf1b722bc2d41507fa4f24694bebd68dcfa7c7957a9a7891dfd"
Oct 30 05:39:20.302803 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:20.302775    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb" event={"ID":"8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f","Type":"ContainerDied","Data":"48bcaf55176641eb56e74bd0f68811fca0c92c1e9e04c44d1ef10666cb858a97"}
Oct 30 05:39:20.302871 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:20.302805    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb"
Oct 30 05:39:20.311614 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:20.311597    5898 scope.go:122] "RemoveContainer" containerID="98c683bf5f41dcf1b722bc2d41507fa4f24694bebd68dcfa7c7957a9a7891dfd"
Oct 30 05:39:20.311873 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:39:20.311852    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98c683bf5f41dcf1b722bc2d41507fa4f24694bebd68dcfa7c7957a9a7891dfd\": container with ID starting with 98c683bf5f41dcf1b722bc2d41507fa4f24694bebd68dcfa7c7957a9a7891dfd not found: ID does not exist" containerID="98c683bf5f41dcf1b722bc2d41507fa4f24694bebd68dcfa7c7957a9a7891dfd"
Oct 30 05:39:20.312011 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:20.311879    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98c683bf5f41dcf1b722bc2d41507fa4f24694bebd68dcfa7c7957a9a7891dfd"} err="failed to get container status \"98c683bf5f41dcf1b722bc2d41507fa4f24694bebd68dcfa7c7957a9a7891dfd\": rpc error: code = NotFound desc = could not find container \"98c683bf5f41dcf1b722bc2d41507fa4f24694bebd68dcfa7c7957a9a7891dfd\": container with ID starting with 98c683bf5f41dcf1b722bc2d41507fa4f24694bebd68dcfa7c7957a9a7891dfd not found: ID does not exist"
Oct 30 05:39:20.312011 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:20.311934    5898 scope.go:122] "RemoveContainer" containerID="f41fe135137090ccc037d32facef08d38129a7fcef77289a60839c9df7d540fb"
Oct 30 05:39:20.403832 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:20.402984    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh"]
Oct 30 05:39:20.403832 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:20.403067    5898 patch_prober.go:29] interesting pod/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 05:39:20.403832 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:20.403115    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp" podUID="fa97913c-19e1-432c-9bd2-b1ce3278f7a5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 05:39:20.412112 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:20.412078    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-f8mvh"]
Oct 30 05:39:20.440643 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:20.440530    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb"]
Oct 30 05:39:20.444862 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:20.444831    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-68fjb"]
Oct 30 05:39:20.449052 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:20.448998    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw"]
Oct 30 05:39:20.449406 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:20.449382    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw" podUID="34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd" containerName="istio-proxy" containerID="cri-o://a54c751205acaae25e07f729e2ed2cb3856bc8a87f507a6170a2f74ff6711d44" gracePeriod=30
Oct 30 05:39:20.492845 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:20.492809    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:20.890113 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:20.890082    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f" path="/var/lib/kubelet/pods/8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f/volumes"
Oct 30 05:39:20.890572 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:20.890558    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4dc6435-93bb-4b45-bdd5-3a80868e15e3" path="/var/lib/kubelet/pods/b4dc6435-93bb-4b45-bdd5-3a80868e15e3/volumes"
Oct 30 05:39:22.020346 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:22.020285    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:23.805552 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:23.805511    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:25.076407 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:25.076374    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:25.577069 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:25.577032    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp"
Oct 30 05:39:25.662569 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:25.662539    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa97913c-19e1-432c-9bd2-b1ce3278f7a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa97913c-19e1-432c-9bd2-b1ce3278f7a5-workload-certs\") pod \"fa97913c-19e1-432c-9bd2-b1ce3278f7a5\" (UID: \"fa97913c-19e1-432c-9bd2-b1ce3278f7a5\") "
Oct 30 05:39:25.662769 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:25.662582    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa97913c-19e1-432c-9bd2-b1ce3278f7a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa97913c-19e1-432c-9bd2-b1ce3278f7a5-credential-socket\") pod \"fa97913c-19e1-432c-9bd2-b1ce3278f7a5\" (UID: \"fa97913c-19e1-432c-9bd2-b1ce3278f7a5\") "
Oct 30 05:39:25.662769 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:25.662668    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa97913c-19e1-432c-9bd2-b1ce3278f7a5-istio-token\" (UniqueName: \"kubernetes.io/projected/fa97913c-19e1-432c-9bd2-b1ce3278f7a5-istio-token\") pod \"fa97913c-19e1-432c-9bd2-b1ce3278f7a5\" (UID: \"fa97913c-19e1-432c-9bd2-b1ce3278f7a5\") "
Oct 30 05:39:25.662769 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:25.662702    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa97913c-19e1-432c-9bd2-b1ce3278f7a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa97913c-19e1-432c-9bd2-b1ce3278f7a5-istio-data\") pod \"fa97913c-19e1-432c-9bd2-b1ce3278f7a5\" (UID: \"fa97913c-19e1-432c-9bd2-b1ce3278f7a5\") "
Oct 30 05:39:25.662769 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:25.662738    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa97913c-19e1-432c-9bd2-b1ce3278f7a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa97913c-19e1-432c-9bd2-b1ce3278f7a5-istiod-ca-cert\") pod \"fa97913c-19e1-432c-9bd2-b1ce3278f7a5\" (UID: \"fa97913c-19e1-432c-9bd2-b1ce3278f7a5\") "
Oct 30 05:39:25.663020 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:25.662870    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa97913c-19e1-432c-9bd2-b1ce3278f7a5-kube-api-access-7448q\" (UniqueName: \"kubernetes.io/projected/fa97913c-19e1-432c-9bd2-b1ce3278f7a5-kube-api-access-7448q\") pod \"fa97913c-19e1-432c-9bd2-b1ce3278f7a5\" (UID: \"fa97913c-19e1-432c-9bd2-b1ce3278f7a5\") "
Oct 30 05:39:25.663020 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:25.662942    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa97913c-19e1-432c-9bd2-b1ce3278f7a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa97913c-19e1-432c-9bd2-b1ce3278f7a5-workload-socket\") pod \"fa97913c-19e1-432c-9bd2-b1ce3278f7a5\" (UID: \"fa97913c-19e1-432c-9bd2-b1ce3278f7a5\") "
Oct 30 05:39:25.663020 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:25.662971    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa97913c-19e1-432c-9bd2-b1ce3278f7a5-credential-socket" pod "fa97913c-19e1-432c-9bd2-b1ce3278f7a5" (UID: "fa97913c-19e1-432c-9bd2-b1ce3278f7a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:39:25.663020 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:25.662999    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa97913c-19e1-432c-9bd2-b1ce3278f7a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa97913c-19e1-432c-9bd2-b1ce3278f7a5-istio-podinfo\") pod \"fa97913c-19e1-432c-9bd2-b1ce3278f7a5\" (UID: \"fa97913c-19e1-432c-9bd2-b1ce3278f7a5\") "
Oct 30 05:39:25.663227 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:25.663066    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa97913c-19e1-432c-9bd2-b1ce3278f7a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa97913c-19e1-432c-9bd2-b1ce3278f7a5-istio-envoy\") pod \"fa97913c-19e1-432c-9bd2-b1ce3278f7a5\" (UID: \"fa97913c-19e1-432c-9bd2-b1ce3278f7a5\") "
Oct 30 05:39:25.663349 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:25.663326    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa97913c-19e1-432c-9bd2-b1ce3278f7a5-istiod-ca-cert" pod "fa97913c-19e1-432c-9bd2-b1ce3278f7a5" (UID: "fa97913c-19e1-432c-9bd2-b1ce3278f7a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:39:25.663456 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:25.663435    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa97913c-19e1-432c-9bd2-b1ce3278f7a5-workload-socket" pod "fa97913c-19e1-432c-9bd2-b1ce3278f7a5" (UID: "fa97913c-19e1-432c-9bd2-b1ce3278f7a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:39:25.663759 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:25.663534    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa97913c-19e1-432c-9bd2-b1ce3278f7a5-istio-data" pod "fa97913c-19e1-432c-9bd2-b1ce3278f7a5" (UID: "fa97913c-19e1-432c-9bd2-b1ce3278f7a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:39:25.664266 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:25.664239    5898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa97913c-19e1-432c-9bd2-b1ce3278f7a5-credential-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:25.664368 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:25.664271    5898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa97913c-19e1-432c-9bd2-b1ce3278f7a5-istio-data\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:25.664368 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:25.664288    5898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa97913c-19e1-432c-9bd2-b1ce3278f7a5-istiod-ca-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:25.664368 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:25.664303    5898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa97913c-19e1-432c-9bd2-b1ce3278f7a5-workload-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:25.664556 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:25.664373    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa97913c-19e1-432c-9bd2-b1ce3278f7a5-workload-certs" pod "fa97913c-19e1-432c-9bd2-b1ce3278f7a5" (UID: "fa97913c-19e1-432c-9bd2-b1ce3278f7a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:39:25.666224 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:25.666198    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa97913c-19e1-432c-9bd2-b1ce3278f7a5-istio-podinfo" pod "fa97913c-19e1-432c-9bd2-b1ce3278f7a5" (UID: "fa97913c-19e1-432c-9bd2-b1ce3278f7a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:39:25.666224 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:25.666202    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa97913c-19e1-432c-9bd2-b1ce3278f7a5-kube-api-access-7448q" pod "fa97913c-19e1-432c-9bd2-b1ce3278f7a5" (UID: "fa97913c-19e1-432c-9bd2-b1ce3278f7a5"). InnerVolumeSpecName "kube-api-access-7448q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:39:25.666342 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:25.666222    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa97913c-19e1-432c-9bd2-b1ce3278f7a5-istio-token" pod "fa97913c-19e1-432c-9bd2-b1ce3278f7a5" (UID: "fa97913c-19e1-432c-9bd2-b1ce3278f7a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:39:25.667092 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:25.667065    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa97913c-19e1-432c-9bd2-b1ce3278f7a5-istio-envoy" pod "fa97913c-19e1-432c-9bd2-b1ce3278f7a5" (UID: "fa97913c-19e1-432c-9bd2-b1ce3278f7a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:39:25.689095 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:25.689045    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:25.692039 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:25.692019    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw"
Oct 30 05:39:25.764885 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:25.764856    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd-istio-data\") pod \"34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd\" (UID: \"34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd\") "
Oct 30 05:39:25.765097 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:25.764950    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd-istiod-ca-cert\") pod \"34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd\" (UID: \"34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd\") "
Oct 30 05:39:25.765097 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:25.765013    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd-istio-token\" (UniqueName: \"kubernetes.io/projected/34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd-istio-token\") pod \"34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd\" (UID: \"34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd\") "
Oct 30 05:39:25.765097 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:25.765073    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd-workload-socket\") pod \"34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd\" (UID: \"34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd\") "
Oct 30 05:39:25.765282 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:25.765143    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd-credential-socket\") pod \"34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd\" (UID: \"34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd\") "
Oct 30 05:39:25.765282 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:25.765189    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd-istio-podinfo\") pod \"34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd\" (UID: \"34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd\") "
Oct 30 05:39:25.765282 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:25.765247    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd-kube-api-access-gxgbk\" (UniqueName: \"kubernetes.io/projected/34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd-kube-api-access-gxgbk\") pod \"34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd\" (UID: \"34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd\") "
Oct 30 05:39:25.765441 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:25.765281    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd-workload-certs\") pod \"34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd\" (UID: \"34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd\") "
Oct 30 05:39:25.765441 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:25.765308    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd-istiod-ca-cert" pod "34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd" (UID: "34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:39:25.765441 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:25.765315    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd-istio-envoy\") pod \"34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd\" (UID: \"34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd\") "
Oct 30 05:39:25.765711 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:25.765685    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd-workload-socket" pod "34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd" (UID: "34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:39:25.765884 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:25.765172    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd-istio-data" pod "34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd" (UID: "34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:39:25.765994 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:25.765895    5898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd-workload-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:25.765994 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:25.765936    5898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa97913c-19e1-432c-9bd2-b1ce3278f7a5-istio-token\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:25.765994 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:25.765951    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7448q\" (UniqueName: \"kubernetes.io/projected/fa97913c-19e1-432c-9bd2-b1ce3278f7a5-kube-api-access-7448q\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:25.765994 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:25.765959    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd-credential-socket" pod "34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd" (UID: "34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:39:25.765994 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:25.765969    5898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa97913c-19e1-432c-9bd2-b1ce3278f7a5-istio-podinfo\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:25.766282 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:25.766003    5898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd-istiod-ca-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:25.766282 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:25.766020    5898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa97913c-19e1-432c-9bd2-b1ce3278f7a5-istio-envoy\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:25.766282 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:25.766034    5898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa97913c-19e1-432c-9bd2-b1ce3278f7a5-workload-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:25.766282 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:25.766111    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd-workload-certs" pod "34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd" (UID: "34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:39:25.767420 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:25.767391    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd-istio-token" pod "34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd" (UID: "34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:39:25.767756 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:25.767735    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd-istio-podinfo" pod "34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd" (UID: "34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:39:25.767857 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:25.767739    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd-istio-envoy" pod "34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd" (UID: "34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:39:25.768053 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:25.768034    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd-kube-api-access-gxgbk" pod "34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd" (UID: "34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd"). InnerVolumeSpecName "kube-api-access-gxgbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:39:25.867237 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:25.867205    5898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd-credential-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:25.867237 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:25.867241    5898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd-istio-podinfo\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:25.867420 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:25.867258    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxgbk\" (UniqueName: \"kubernetes.io/projected/34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd-kube-api-access-gxgbk\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:25.867420 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:25.867274    5898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd-workload-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:25.867420 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:25.867290    5898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd-istio-envoy\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:25.867420 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:25.867301    5898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd-istio-data\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:25.867420 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:25.867312    5898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd-istio-token\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:26.340033 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:26.339997    5898 generic.go:356] "Generic (PLEG): container finished" podID="34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd" containerID="a54c751205acaae25e07f729e2ed2cb3856bc8a87f507a6170a2f74ff6711d44" exitCode=0
Oct 30 05:39:26.340428 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:26.340046    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw"
Oct 30 05:39:26.340428 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:26.340062    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw" event={"ID":"34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd","Type":"ContainerDied","Data":"a54c751205acaae25e07f729e2ed2cb3856bc8a87f507a6170a2f74ff6711d44"}
Oct 30 05:39:26.340428 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:26.340120    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw" event={"ID":"34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd","Type":"ContainerDied","Data":"106b82f84b7f1734e762195e7e00dd4b6246a7eefe7399c8c0f8b7def44e5ef2"}
Oct 30 05:39:26.340428 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:26.340150    5898 scope.go:122] "RemoveContainer" containerID="a54c751205acaae25e07f729e2ed2cb3856bc8a87f507a6170a2f74ff6711d44"
Oct 30 05:39:26.341495 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:26.341472    5898 generic.go:356] "Generic (PLEG): container finished" podID="fa97913c-19e1-432c-9bd2-b1ce3278f7a5" containerID="0d431a2552cdab88b3e21fa66e0a7a7321fbc890b500e7a693d010be9e80209a" exitCode=0
Oct 30 05:39:26.341610 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:26.341502    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp" event={"ID":"fa97913c-19e1-432c-9bd2-b1ce3278f7a5","Type":"ContainerDied","Data":"0d431a2552cdab88b3e21fa66e0a7a7321fbc890b500e7a693d010be9e80209a"}
Oct 30 05:39:26.341610 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:26.341534    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp"
Oct 30 05:39:26.341610 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:26.341544    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp" event={"ID":"fa97913c-19e1-432c-9bd2-b1ce3278f7a5","Type":"ContainerDied","Data":"e844e208aed5dfaa602fb132cef5599212b3fd9cf2811aa0946e61f1f1b555a7"}
Oct 30 05:39:26.349893 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:26.349875    5898 scope.go:122] "RemoveContainer" containerID="a54c751205acaae25e07f729e2ed2cb3856bc8a87f507a6170a2f74ff6711d44"
Oct 30 05:39:26.350251 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:39:26.350230    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a54c751205acaae25e07f729e2ed2cb3856bc8a87f507a6170a2f74ff6711d44\": container with ID starting with a54c751205acaae25e07f729e2ed2cb3856bc8a87f507a6170a2f74ff6711d44 not found: ID does not exist" containerID="a54c751205acaae25e07f729e2ed2cb3856bc8a87f507a6170a2f74ff6711d44"
Oct 30 05:39:26.350330 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:26.350258    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a54c751205acaae25e07f729e2ed2cb3856bc8a87f507a6170a2f74ff6711d44"} err="failed to get container status \"a54c751205acaae25e07f729e2ed2cb3856bc8a87f507a6170a2f74ff6711d44\": rpc error: code = NotFound desc = could not find container \"a54c751205acaae25e07f729e2ed2cb3856bc8a87f507a6170a2f74ff6711d44\": container with ID starting with a54c751205acaae25e07f729e2ed2cb3856bc8a87f507a6170a2f74ff6711d44 not found: ID does not exist"
Oct 30 05:39:26.350330 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:26.350276    5898 scope.go:122] "RemoveContainer" containerID="0d431a2552cdab88b3e21fa66e0a7a7321fbc890b500e7a693d010be9e80209a"
Oct 30 05:39:26.359078 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:26.359060    5898 scope.go:122] "RemoveContainer" containerID="0d431a2552cdab88b3e21fa66e0a7a7321fbc890b500e7a693d010be9e80209a"
Oct 30 05:39:26.359329 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:39:26.359312    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d431a2552cdab88b3e21fa66e0a7a7321fbc890b500e7a693d010be9e80209a\": container with ID starting with 0d431a2552cdab88b3e21fa66e0a7a7321fbc890b500e7a693d010be9e80209a not found: ID does not exist" containerID="0d431a2552cdab88b3e21fa66e0a7a7321fbc890b500e7a693d010be9e80209a"
Oct 30 05:39:26.359377 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:26.359337    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d431a2552cdab88b3e21fa66e0a7a7321fbc890b500e7a693d010be9e80209a"} err="failed to get container status \"0d431a2552cdab88b3e21fa66e0a7a7321fbc890b500e7a693d010be9e80209a\": rpc error: code = NotFound desc = could not find container \"0d431a2552cdab88b3e21fa66e0a7a7321fbc890b500e7a693d010be9e80209a\": container with ID starting with 0d431a2552cdab88b3e21fa66e0a7a7321fbc890b500e7a693d010be9e80209a not found: ID does not exist"
Oct 30 05:39:26.391047 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:26.391021    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw"]
Oct 30 05:39:26.402173 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:26.402145    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-34722e-openshift-default-5dd74b4f95-84ngw"]
Oct 30 05:39:26.437688 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:26.437664    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp"]
Oct 30 05:39:26.441623 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:26.441596    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-397179-openshift-default-bc795b5d9-g6gkp"]
Oct 30 05:39:26.821756 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:39:26.821722    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:39:26.893233 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:26.890756    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd" path="/var/lib/kubelet/pods/34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd/volumes"
Oct 30 05:39:26.893233 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:26.891548    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa97913c-19e1-432c-9bd2-b1ce3278f7a5" path="/var/lib/kubelet/pods/fa97913c-19e1-432c-9bd2-b1ce3278f7a5/volumes"
Oct 30 05:39:27.417179 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:27.417143    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:29.284272 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:29.284238    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:29.892735 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:29.892705    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:30.006037 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:30.005971    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:30.985128 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:30.985093    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:31.673186 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:31.673142    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:31.793355 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:31.793321    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:32.860588 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:32.860555    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:33.492041 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:33.492007    5898 scope.go:122] "RemoveContainer" containerID="2aafcc2a4818acbb0e27c615686c629d50d01edc3a9035638051e885ab4e7e82"
Oct 30 05:39:34.345211 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:34.345172    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:34.956461 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:34.956405    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:35.073384 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:35.073333    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:35.175916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:35.175847    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:35.642113 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:35.642079    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:36.875061 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:36.875023    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:38.448438 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:38.448401    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:38.845154 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:38.845114    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:39:38.974771 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:38.974735    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:39.133387 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:39.133252    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:39.643777 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:39.643736    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:41.173317 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:41.173285    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:42.460333 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:42.460301    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:42.576218 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:42.576164    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:42.813745 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:42.813658    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:43.068121 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:43.068007    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:43.272815 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:43.272780    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:44.188560 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:44.188526    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:44.306991 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:44.306961    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:45.054372 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:45.054344    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:46.241184 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:46.241157    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:46.346833 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:46.346801    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:46.591632 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:46.591601    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85"]
Oct 30 05:39:46.591925 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:46.591863    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85" podUID="96f0bc26-fbf5-4d4c-a146-207dc12a4e20" containerName="istio-proxy" containerID="cri-o://9eeba5e3797d862b575859206e238d2cb92adc2465f585531673e335198cec8b" gracePeriod=30
Oct 30 05:39:46.612304 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:46.612278    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:46.933180 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:46.933084    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:48.317441 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:48.317402    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-0b21ba/distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6"]
Oct 30 05:39:48.329489 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:48.329457    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-0b21ba/distinct-w5-0b21ba-kserve-7674c59666-m2sbd"]
Oct 30 05:39:48.333914 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:48.333880    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849"]
Oct 30 05:39:48.436148 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:48.436112    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:48.965858 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:48.965822    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:50.102279 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:50.102243    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:39:50.114954 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:50.114924    5898 patch_prober.go:29] interesting pod/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 05:39:50.115105 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:50.114975    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85" podUID="96f0bc26-fbf5-4d4c-a146-207dc12a4e20" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 05:39:50.896169 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:50.896124    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:51.067929 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:51.067855    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq"]
Oct 30 05:39:51.068296 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:51.068265    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq" podUID="3b386772-86df-4d1c-a5c4-fc27901eb9fd" containerName="istio-proxy" containerID="cri-o://4d3d1302c993c1ba8acdb8a90fd594be0ea096410e51f772f396b3dccd3e69e8" gracePeriod=30
Oct 30 05:39:51.862528 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:51.862473    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85"
Oct 30 05:39:51.962354 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:51.962318    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96f0bc26-fbf5-4d4c-a146-207dc12a4e20-kube-api-access-f9f26\" (UniqueName: \"kubernetes.io/projected/96f0bc26-fbf5-4d4c-a146-207dc12a4e20-kube-api-access-f9f26\") pod \"96f0bc26-fbf5-4d4c-a146-207dc12a4e20\" (UID: \"96f0bc26-fbf5-4d4c-a146-207dc12a4e20\") "
Oct 30 05:39:51.962544 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:51.962430    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/96f0bc26-fbf5-4d4c-a146-207dc12a4e20-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96f0bc26-fbf5-4d4c-a146-207dc12a4e20-istio-podinfo\") pod \"96f0bc26-fbf5-4d4c-a146-207dc12a4e20\" (UID: \"96f0bc26-fbf5-4d4c-a146-207dc12a4e20\") "
Oct 30 05:39:51.962544 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:51.962480    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96f0bc26-fbf5-4d4c-a146-207dc12a4e20-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96f0bc26-fbf5-4d4c-a146-207dc12a4e20-workload-certs\") pod \"96f0bc26-fbf5-4d4c-a146-207dc12a4e20\" (UID: \"96f0bc26-fbf5-4d4c-a146-207dc12a4e20\") "
Oct 30 05:39:51.962544 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:51.962528    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96f0bc26-fbf5-4d4c-a146-207dc12a4e20-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96f0bc26-fbf5-4d4c-a146-207dc12a4e20-istiod-ca-cert\") pod \"96f0bc26-fbf5-4d4c-a146-207dc12a4e20\" (UID: \"96f0bc26-fbf5-4d4c-a146-207dc12a4e20\") "
Oct 30 05:39:51.962731 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:51.962570    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96f0bc26-fbf5-4d4c-a146-207dc12a4e20-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96f0bc26-fbf5-4d4c-a146-207dc12a4e20-workload-socket\") pod \"96f0bc26-fbf5-4d4c-a146-207dc12a4e20\" (UID: \"96f0bc26-fbf5-4d4c-a146-207dc12a4e20\") "
Oct 30 05:39:51.962731 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:51.962619    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96f0bc26-fbf5-4d4c-a146-207dc12a4e20-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96f0bc26-fbf5-4d4c-a146-207dc12a4e20-credential-socket\") pod \"96f0bc26-fbf5-4d4c-a146-207dc12a4e20\" (UID: \"96f0bc26-fbf5-4d4c-a146-207dc12a4e20\") "
Oct 30 05:39:51.962731 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:51.962652    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96f0bc26-fbf5-4d4c-a146-207dc12a4e20-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96f0bc26-fbf5-4d4c-a146-207dc12a4e20-istio-envoy\") pod \"96f0bc26-fbf5-4d4c-a146-207dc12a4e20\" (UID: \"96f0bc26-fbf5-4d4c-a146-207dc12a4e20\") "
Oct 30 05:39:51.962731 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:51.962710    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96f0bc26-fbf5-4d4c-a146-207dc12a4e20-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96f0bc26-fbf5-4d4c-a146-207dc12a4e20-istio-data\") pod \"96f0bc26-fbf5-4d4c-a146-207dc12a4e20\" (UID: \"96f0bc26-fbf5-4d4c-a146-207dc12a4e20\") "
Oct 30 05:39:51.962976 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:51.962740    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96f0bc26-fbf5-4d4c-a146-207dc12a4e20-istio-token\" (UniqueName: \"kubernetes.io/projected/96f0bc26-fbf5-4d4c-a146-207dc12a4e20-istio-token\") pod \"96f0bc26-fbf5-4d4c-a146-207dc12a4e20\" (UID: \"96f0bc26-fbf5-4d4c-a146-207dc12a4e20\") "
Oct 30 05:39:51.962976 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:51.962825    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96f0bc26-fbf5-4d4c-a146-207dc12a4e20-workload-certs" pod "96f0bc26-fbf5-4d4c-a146-207dc12a4e20" (UID: "96f0bc26-fbf5-4d4c-a146-207dc12a4e20"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:39:51.963127 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:51.963104    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96f0bc26-fbf5-4d4c-a146-207dc12a4e20-workload-socket" pod "96f0bc26-fbf5-4d4c-a146-207dc12a4e20" (UID: "96f0bc26-fbf5-4d4c-a146-207dc12a4e20"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:39:51.963202 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:51.963150    5898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96f0bc26-fbf5-4d4c-a146-207dc12a4e20-workload-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:51.963202 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:51.963165    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96f0bc26-fbf5-4d4c-a146-207dc12a4e20-istiod-ca-cert" pod "96f0bc26-fbf5-4d4c-a146-207dc12a4e20" (UID: "96f0bc26-fbf5-4d4c-a146-207dc12a4e20"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:39:51.963452 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:51.963428    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96f0bc26-fbf5-4d4c-a146-207dc12a4e20-istio-data" pod "96f0bc26-fbf5-4d4c-a146-207dc12a4e20" (UID: "96f0bc26-fbf5-4d4c-a146-207dc12a4e20"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:39:51.963537 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:51.963460    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96f0bc26-fbf5-4d4c-a146-207dc12a4e20-credential-socket" pod "96f0bc26-fbf5-4d4c-a146-207dc12a4e20" (UID: "96f0bc26-fbf5-4d4c-a146-207dc12a4e20"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:39:51.964775 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:51.964746    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96f0bc26-fbf5-4d4c-a146-207dc12a4e20-kube-api-access-f9f26" pod "96f0bc26-fbf5-4d4c-a146-207dc12a4e20" (UID: "96f0bc26-fbf5-4d4c-a146-207dc12a4e20"). InnerVolumeSpecName "kube-api-access-f9f26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:39:51.964975 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:51.964957    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/96f0bc26-fbf5-4d4c-a146-207dc12a4e20-istio-podinfo" pod "96f0bc26-fbf5-4d4c-a146-207dc12a4e20" (UID: "96f0bc26-fbf5-4d4c-a146-207dc12a4e20"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:39:51.965262 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:51.965238    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96f0bc26-fbf5-4d4c-a146-207dc12a4e20-istio-envoy" pod "96f0bc26-fbf5-4d4c-a146-207dc12a4e20" (UID: "96f0bc26-fbf5-4d4c-a146-207dc12a4e20"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:39:51.965576 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:51.965557    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96f0bc26-fbf5-4d4c-a146-207dc12a4e20-istio-token" pod "96f0bc26-fbf5-4d4c-a146-207dc12a4e20" (UID: "96f0bc26-fbf5-4d4c-a146-207dc12a4e20"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:39:52.063965 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:52.063933    5898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96f0bc26-fbf5-4d4c-a146-207dc12a4e20-istio-data\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:52.063965 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:52.063958    5898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96f0bc26-fbf5-4d4c-a146-207dc12a4e20-istio-token\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:52.063965 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:52.063970    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9f26\" (UniqueName: \"kubernetes.io/projected/96f0bc26-fbf5-4d4c-a146-207dc12a4e20-kube-api-access-f9f26\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:52.064278 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:52.063984    5898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96f0bc26-fbf5-4d4c-a146-207dc12a4e20-istio-podinfo\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:52.064278 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:52.064001    5898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96f0bc26-fbf5-4d4c-a146-207dc12a4e20-istiod-ca-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:52.064278 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:52.064009    5898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96f0bc26-fbf5-4d4c-a146-207dc12a4e20-workload-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:52.064278 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:52.064016    5898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96f0bc26-fbf5-4d4c-a146-207dc12a4e20-credential-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:52.064278 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:52.064024    5898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96f0bc26-fbf5-4d4c-a146-207dc12a4e20-istio-envoy\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:52.494058 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:52.494020    5898 generic.go:356] "Generic (PLEG): container finished" podID="96f0bc26-fbf5-4d4c-a146-207dc12a4e20" containerID="9eeba5e3797d862b575859206e238d2cb92adc2465f585531673e335198cec8b" exitCode=0
Oct 30 05:39:52.494265 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:52.494058    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85" event={"ID":"96f0bc26-fbf5-4d4c-a146-207dc12a4e20","Type":"ContainerDied","Data":"9eeba5e3797d862b575859206e238d2cb92adc2465f585531673e335198cec8b"}
Oct 30 05:39:52.494265 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:52.494070    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85"
Oct 30 05:39:52.494265 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:52.494113    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85" event={"ID":"96f0bc26-fbf5-4d4c-a146-207dc12a4e20","Type":"ContainerDied","Data":"8a6c8bbb55d8e7ab076ba6588f13424624eed042e1684d8f0c134a712e34a544"}
Oct 30 05:39:52.494265 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:52.494135    5898 scope.go:122] "RemoveContainer" containerID="9eeba5e3797d862b575859206e238d2cb92adc2465f585531673e335198cec8b"
Oct 30 05:39:52.505661 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:52.505638    5898 scope.go:122] "RemoveContainer" containerID="9eeba5e3797d862b575859206e238d2cb92adc2465f585531673e335198cec8b"
Oct 30 05:39:52.505967 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:39:52.505945    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9eeba5e3797d862b575859206e238d2cb92adc2465f585531673e335198cec8b\": container with ID starting with 9eeba5e3797d862b575859206e238d2cb92adc2465f585531673e335198cec8b not found: ID does not exist" containerID="9eeba5e3797d862b575859206e238d2cb92adc2465f585531673e335198cec8b"
Oct 30 05:39:52.506028 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:52.505975    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9eeba5e3797d862b575859206e238d2cb92adc2465f585531673e335198cec8b"} err="failed to get container status \"9eeba5e3797d862b575859206e238d2cb92adc2465f585531673e335198cec8b\": rpc error: code = NotFound desc = could not find container \"9eeba5e3797d862b575859206e238d2cb92adc2465f585531673e335198cec8b\": container with ID starting with 9eeba5e3797d862b575859206e238d2cb92adc2465f585531673e335198cec8b not found: ID does not exist"
Oct 30 05:39:52.557759 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:52.557734    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85"]
Oct 30 05:39:52.569058 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:52.569025    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-35a94e-openshift-default-586cdc6fdc-hvl85"]
Oct 30 05:39:52.685926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:52.685872    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:52.889499 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:52.889468    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96f0bc26-fbf5-4d4c-a146-207dc12a4e20" path="/var/lib/kubelet/pods/96f0bc26-fbf5-4d4c-a146-207dc12a4e20/volumes"
Oct 30 05:39:54.067825 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:54.067776    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:54.179331 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:54.179204    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:54.496563 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:54.496517    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:56.279338 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:56.279277    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:56.334283 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:56.333867    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq"
Oct 30 05:39:56.407052 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:56.407021    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b386772-86df-4d1c-a5c4-fc27901eb9fd-istio-token\" (UniqueName: \"kubernetes.io/projected/3b386772-86df-4d1c-a5c4-fc27901eb9fd-istio-token\") pod \"3b386772-86df-4d1c-a5c4-fc27901eb9fd\" (UID: \"3b386772-86df-4d1c-a5c4-fc27901eb9fd\") "
Oct 30 05:39:56.407271 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:56.407072    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b386772-86df-4d1c-a5c4-fc27901eb9fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b386772-86df-4d1c-a5c4-fc27901eb9fd-istio-data\") pod \"3b386772-86df-4d1c-a5c4-fc27901eb9fd\" (UID: \"3b386772-86df-4d1c-a5c4-fc27901eb9fd\") "
Oct 30 05:39:56.407271 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:56.407102    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b386772-86df-4d1c-a5c4-fc27901eb9fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b386772-86df-4d1c-a5c4-fc27901eb9fd-istiod-ca-cert\") pod \"3b386772-86df-4d1c-a5c4-fc27901eb9fd\" (UID: \"3b386772-86df-4d1c-a5c4-fc27901eb9fd\") "
Oct 30 05:39:56.407271 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:56.407125    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b386772-86df-4d1c-a5c4-fc27901eb9fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b386772-86df-4d1c-a5c4-fc27901eb9fd-credential-socket\") pod \"3b386772-86df-4d1c-a5c4-fc27901eb9fd\" (UID: \"3b386772-86df-4d1c-a5c4-fc27901eb9fd\") "
Oct 30 05:39:56.407271 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:56.407156    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b386772-86df-4d1c-a5c4-fc27901eb9fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b386772-86df-4d1c-a5c4-fc27901eb9fd-workload-certs\") pod \"3b386772-86df-4d1c-a5c4-fc27901eb9fd\" (UID: \"3b386772-86df-4d1c-a5c4-fc27901eb9fd\") "
Oct 30 05:39:56.407271 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:56.407228    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b386772-86df-4d1c-a5c4-fc27901eb9fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b386772-86df-4d1c-a5c4-fc27901eb9fd-workload-socket\") pod \"3b386772-86df-4d1c-a5c4-fc27901eb9fd\" (UID: \"3b386772-86df-4d1c-a5c4-fc27901eb9fd\") "
Oct 30 05:39:56.407271 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:56.407266    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b386772-86df-4d1c-a5c4-fc27901eb9fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b386772-86df-4d1c-a5c4-fc27901eb9fd-istio-podinfo\") pod \"3b386772-86df-4d1c-a5c4-fc27901eb9fd\" (UID: \"3b386772-86df-4d1c-a5c4-fc27901eb9fd\") "
Oct 30 05:39:56.407645 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:56.407311    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b386772-86df-4d1c-a5c4-fc27901eb9fd-kube-api-access-5mwss\" (UniqueName: \"kubernetes.io/projected/3b386772-86df-4d1c-a5c4-fc27901eb9fd-kube-api-access-5mwss\") pod \"3b386772-86df-4d1c-a5c4-fc27901eb9fd\" (UID: \"3b386772-86df-4d1c-a5c4-fc27901eb9fd\") "
Oct 30 05:39:56.407645 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:56.407377    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b386772-86df-4d1c-a5c4-fc27901eb9fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b386772-86df-4d1c-a5c4-fc27901eb9fd-istio-envoy\") pod \"3b386772-86df-4d1c-a5c4-fc27901eb9fd\" (UID: \"3b386772-86df-4d1c-a5c4-fc27901eb9fd\") "
Oct 30 05:39:56.407755 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:56.407695    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b386772-86df-4d1c-a5c4-fc27901eb9fd-credential-socket" pod "3b386772-86df-4d1c-a5c4-fc27901eb9fd" (UID: "3b386772-86df-4d1c-a5c4-fc27901eb9fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:39:56.407755 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:56.407709    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b386772-86df-4d1c-a5c4-fc27901eb9fd-workload-socket" pod "3b386772-86df-4d1c-a5c4-fc27901eb9fd" (UID: "3b386772-86df-4d1c-a5c4-fc27901eb9fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:39:56.407880 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:56.407818    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b386772-86df-4d1c-a5c4-fc27901eb9fd-istio-data" pod "3b386772-86df-4d1c-a5c4-fc27901eb9fd" (UID: "3b386772-86df-4d1c-a5c4-fc27901eb9fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:39:56.407880 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:56.407845    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b386772-86df-4d1c-a5c4-fc27901eb9fd-workload-certs" pod "3b386772-86df-4d1c-a5c4-fc27901eb9fd" (UID: "3b386772-86df-4d1c-a5c4-fc27901eb9fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:39:56.408026 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:56.408010    5898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b386772-86df-4d1c-a5c4-fc27901eb9fd-workload-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:56.408083 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:56.408028    5898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b386772-86df-4d1c-a5c4-fc27901eb9fd-istio-data\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:56.408083 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:56.408070    5898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b386772-86df-4d1c-a5c4-fc27901eb9fd-credential-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:56.408189 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:56.408086    5898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b386772-86df-4d1c-a5c4-fc27901eb9fd-workload-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:56.408246 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:56.408195    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b386772-86df-4d1c-a5c4-fc27901eb9fd-istiod-ca-cert" pod "3b386772-86df-4d1c-a5c4-fc27901eb9fd" (UID: "3b386772-86df-4d1c-a5c4-fc27901eb9fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:39:56.416442 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:56.409869    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b386772-86df-4d1c-a5c4-fc27901eb9fd-istio-token" pod "3b386772-86df-4d1c-a5c4-fc27901eb9fd" (UID: "3b386772-86df-4d1c-a5c4-fc27901eb9fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:39:56.416442 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:56.410026    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b386772-86df-4d1c-a5c4-fc27901eb9fd-kube-api-access-5mwss" pod "3b386772-86df-4d1c-a5c4-fc27901eb9fd" (UID: "3b386772-86df-4d1c-a5c4-fc27901eb9fd"). InnerVolumeSpecName "kube-api-access-5mwss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:39:56.416442 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:56.410161    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b386772-86df-4d1c-a5c4-fc27901eb9fd-istio-envoy" pod "3b386772-86df-4d1c-a5c4-fc27901eb9fd" (UID: "3b386772-86df-4d1c-a5c4-fc27901eb9fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:39:56.416442 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:56.410174    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b386772-86df-4d1c-a5c4-fc27901eb9fd-istio-podinfo" pod "3b386772-86df-4d1c-a5c4-fc27901eb9fd" (UID: "3b386772-86df-4d1c-a5c4-fc27901eb9fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:39:56.508833 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:56.508802    5898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b386772-86df-4d1c-a5c4-fc27901eb9fd-istio-envoy\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:56.508833 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:56.508830    5898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b386772-86df-4d1c-a5c4-fc27901eb9fd-istio-token\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:56.508833 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:56.508839    5898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b386772-86df-4d1c-a5c4-fc27901eb9fd-istiod-ca-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:56.509085 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:56.508849    5898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b386772-86df-4d1c-a5c4-fc27901eb9fd-istio-podinfo\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:56.509085 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:56.508858    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mwss\" (UniqueName: \"kubernetes.io/projected/3b386772-86df-4d1c-a5c4-fc27901eb9fd-kube-api-access-5mwss\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:39:56.519442 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:56.519414    5898 generic.go:356] "Generic (PLEG): container finished" podID="3b386772-86df-4d1c-a5c4-fc27901eb9fd" containerID="4d3d1302c993c1ba8acdb8a90fd594be0ea096410e51f772f396b3dccd3e69e8" exitCode=0
Oct 30 05:39:56.519589 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:56.519463    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq"
Oct 30 05:39:56.519589 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:56.519501    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq" event={"ID":"3b386772-86df-4d1c-a5c4-fc27901eb9fd","Type":"ContainerDied","Data":"4d3d1302c993c1ba8acdb8a90fd594be0ea096410e51f772f396b3dccd3e69e8"}
Oct 30 05:39:56.519589 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:56.519554    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq" event={"ID":"3b386772-86df-4d1c-a5c4-fc27901eb9fd","Type":"ContainerDied","Data":"6791d6fabae85ee6302a7c3c80b4bcc48d552e82148800f90bdee84d5d1077fd"}
Oct 30 05:39:56.519589 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:56.519583    5898 scope.go:122] "RemoveContainer" containerID="4d3d1302c993c1ba8acdb8a90fd594be0ea096410e51f772f396b3dccd3e69e8"
Oct 30 05:39:56.529678 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:56.529661    5898 scope.go:122] "RemoveContainer" containerID="4d3d1302c993c1ba8acdb8a90fd594be0ea096410e51f772f396b3dccd3e69e8"
Oct 30 05:39:56.529963 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:39:56.529943    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d3d1302c993c1ba8acdb8a90fd594be0ea096410e51f772f396b3dccd3e69e8\": container with ID starting with 4d3d1302c993c1ba8acdb8a90fd594be0ea096410e51f772f396b3dccd3e69e8 not found: ID does not exist" containerID="4d3d1302c993c1ba8acdb8a90fd594be0ea096410e51f772f396b3dccd3e69e8"
Oct 30 05:39:56.530010 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:56.529975    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d3d1302c993c1ba8acdb8a90fd594be0ea096410e51f772f396b3dccd3e69e8"} err="failed to get container status \"4d3d1302c993c1ba8acdb8a90fd594be0ea096410e51f772f396b3dccd3e69e8\": rpc error: code = NotFound desc = could not find container \"4d3d1302c993c1ba8acdb8a90fd594be0ea096410e51f772f396b3dccd3e69e8\": container with ID starting with 4d3d1302c993c1ba8acdb8a90fd594be0ea096410e51f772f396b3dccd3e69e8 not found: ID does not exist"
Oct 30 05:39:56.575938 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:56.575916    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq"]
Oct 30 05:39:56.579282 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:56.579261    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-rp8kq"]
Oct 30 05:39:56.817655 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:39:56.817572    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:39:56.891633 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:56.891594    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b386772-86df-4d1c-a5c4-fc27901eb9fd" path="/var/lib/kubelet/pods/3b386772-86df-4d1c-a5c4-fc27901eb9fd/volumes"
Oct 30 05:39:57.760486 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:57.760456    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:57.871434 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:57.871398    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:58.153569 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:58.153536    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:39:59.663777 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.663736    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-157dec48-qpbkd"]
Oct 30 05:39:59.664312 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664292    5898 cpu_manager.go:414] "containerMap: removing container" podUID="47ff272a-ac5c-4f59-bf7b-5c0692d09386" containerName="main"
Oct 30 05:39:59.664390 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664312    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47ff272a-ac5c-4f59-bf7b-5c0692d09386" containerName="main"
Oct 30 05:39:59.664390 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664325    5898 cpu_manager.go:414] "containerMap: removing container" podUID="268c3238-4e5e-4907-b5c0-cf47164701ea" containerName="istio-proxy"
Oct 30 05:39:59.664390 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664332    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="268c3238-4e5e-4907-b5c0-cf47164701ea" containerName="istio-proxy"
Oct 30 05:39:59.664390 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664350    5898 cpu_manager.go:414] "containerMap: removing container" podUID="dc5b5678-4d59-41b9-857b-8c502ec7eb28" containerName="payload-processing"
Oct 30 05:39:59.664390 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664358    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc5b5678-4d59-41b9-857b-8c502ec7eb28" containerName="payload-processing"
Oct 30 05:39:59.664390 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664373    5898 cpu_manager.go:414] "containerMap: removing container" podUID="dfd4e937-c236-4f33-8477-0c55aa536d30" containerName="maas-api"
Oct 30 05:39:59.664390 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664382    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfd4e937-c236-4f33-8477-0c55aa536d30" containerName="maas-api"
Oct 30 05:39:59.664774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664393    5898 cpu_manager.go:414] "containerMap: removing container" podUID="34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd" containerName="istio-proxy"
Oct 30 05:39:59.664774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664400    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd" containerName="istio-proxy"
Oct 30 05:39:59.664774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664414    5898 cpu_manager.go:414] "containerMap: removing container" podUID="4f19c225-09ec-43b9-af12-7c05b7546d72" containerName="payload-pre-processing"
Oct 30 05:39:59.664774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664421    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f19c225-09ec-43b9-af12-7c05b7546d72" containerName="payload-pre-processing"
Oct 30 05:39:59.664774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664432    5898 cpu_manager.go:414] "containerMap: removing container" podUID="b35f68e0-c492-44ba-a145-026a63713b60" containerName="maas-api"
Oct 30 05:39:59.664774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664439    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b35f68e0-c492-44ba-a145-026a63713b60" containerName="maas-api"
Oct 30 05:39:59.664774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664456    5898 cpu_manager.go:414] "containerMap: removing container" podUID="3270ef0e-4775-48c6-8724-baff8e3fac06" containerName="payload-processing"
Oct 30 05:39:59.664774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664466    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3270ef0e-4775-48c6-8724-baff8e3fac06" containerName="payload-processing"
Oct 30 05:39:59.664774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664477    5898 cpu_manager.go:414] "containerMap: removing container" podUID="8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff" containerName="maas-api"
Oct 30 05:39:59.664774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664485    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff" containerName="maas-api"
Oct 30 05:39:59.664774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664499    5898 cpu_manager.go:414] "containerMap: removing container" podUID="4c2b56cf-6fbe-4787-86db-1692ecbb41e0" containerName="istio-proxy"
Oct 30 05:39:59.664774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664507    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c2b56cf-6fbe-4787-86db-1692ecbb41e0" containerName="istio-proxy"
Oct 30 05:39:59.664774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664525    5898 cpu_manager.go:414] "containerMap: removing container" podUID="fa97913c-19e1-432c-9bd2-b1ce3278f7a5" containerName="istio-proxy"
Oct 30 05:39:59.664774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664532    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa97913c-19e1-432c-9bd2-b1ce3278f7a5" containerName="istio-proxy"
Oct 30 05:39:59.664774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664545    5898 cpu_manager.go:414] "containerMap: removing container" podUID="d523008f-c740-4a41-89cb-e82e1ffa0094" containerName="main"
Oct 30 05:39:59.664774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664552    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d523008f-c740-4a41-89cb-e82e1ffa0094" containerName="main"
Oct 30 05:39:59.664774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664563    5898 cpu_manager.go:414] "containerMap: removing container" podUID="38428a53-04c7-400a-9c78-52407e982478" containerName="main"
Oct 30 05:39:59.664774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664570    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38428a53-04c7-400a-9c78-52407e982478" containerName="main"
Oct 30 05:39:59.664774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664586    5898 cpu_manager.go:414] "containerMap: removing container" podUID="a5620415-5516-441d-ab13-3a574d363b59" containerName="istio-proxy"
Oct 30 05:39:59.664774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664593    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5620415-5516-441d-ab13-3a574d363b59" containerName="istio-proxy"
Oct 30 05:39:59.664774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664604    5898 cpu_manager.go:414] "containerMap: removing container" podUID="f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc" containerName="maas-api"
Oct 30 05:39:59.664774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664611    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc" containerName="maas-api"
Oct 30 05:39:59.664774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664622    5898 cpu_manager.go:414] "containerMap: removing container" podUID="5d3b67cf-5656-4d62-8adc-91df8cdcf394" containerName="maas-api"
Oct 30 05:39:59.664774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664629    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d3b67cf-5656-4d62-8adc-91df8cdcf394" containerName="maas-api"
Oct 30 05:39:59.664774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664641    5898 cpu_manager.go:414] "containerMap: removing container" podUID="5e630691-ed49-45ba-ac72-67ef295e80f3" containerName="istio-proxy"
Oct 30 05:39:59.664774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664649    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e630691-ed49-45ba-ac72-67ef295e80f3" containerName="istio-proxy"
Oct 30 05:39:59.664774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664660    5898 cpu_manager.go:414] "containerMap: removing container" podUID="3bc29954-1331-4bdf-9d6c-57e5894a5947" containerName="istio-proxy"
Oct 30 05:39:59.664774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664667    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bc29954-1331-4bdf-9d6c-57e5894a5947" containerName="istio-proxy"
Oct 30 05:39:59.664774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664676    5898 cpu_manager.go:414] "containerMap: removing container" podUID="edceaaa4-f231-474e-85e1-b877ede202ef" containerName="istio-proxy"
Oct 30 05:39:59.664774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664683    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edceaaa4-f231-474e-85e1-b877ede202ef" containerName="istio-proxy"
Oct 30 05:39:59.664774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664692    5898 cpu_manager.go:414] "containerMap: removing container" podUID="c00385be-b173-422b-8ba6-0985c9cf7598" containerName="payload-pre-processing"
Oct 30 05:39:59.664774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664699    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c00385be-b173-422b-8ba6-0985c9cf7598" containerName="payload-pre-processing"
Oct 30 05:39:59.664774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664712    5898 cpu_manager.go:414] "containerMap: removing container" podUID="bfe8c234-c443-44a6-8420-9566768d6d02" containerName="istio-proxy"
Oct 30 05:39:59.664774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664719    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfe8c234-c443-44a6-8420-9566768d6d02" containerName="istio-proxy"
Oct 30 05:39:59.664774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664727    5898 cpu_manager.go:414] "containerMap: removing container" podUID="59cf2966-7359-48ba-bd83-769f0c076397" containerName="istio-proxy"
Oct 30 05:39:59.664774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664735    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59cf2966-7359-48ba-bd83-769f0c076397" containerName="istio-proxy"
Oct 30 05:39:59.664774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664745    5898 cpu_manager.go:414] "containerMap: removing container" podUID="d77a7956-ec67-4cfb-ba98-fd1e6ea6f227" containerName="istio-proxy"
Oct 30 05:39:59.664774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664752    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d77a7956-ec67-4cfb-ba98-fd1e6ea6f227" containerName="istio-proxy"
Oct 30 05:39:59.664774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664761    5898 cpu_manager.go:414] "containerMap: removing container" podUID="8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f" containerName="istio-proxy"
Oct 30 05:39:59.664774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664768    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f" containerName="istio-proxy"
Oct 30 05:39:59.664774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664782    5898 cpu_manager.go:414] "containerMap: removing container" podUID="7063ee78-5b1b-4f4a-b1f0-7898af800db1" containerName="payload-pre-processing"
Oct 30 05:39:59.664774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664789    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7063ee78-5b1b-4f4a-b1f0-7898af800db1" containerName="payload-pre-processing"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664804    5898 cpu_manager.go:414] "containerMap: removing container" podUID="fecce31e-a6fa-443f-aedf-327ef9710f2a" containerName="istio-proxy"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664810    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fecce31e-a6fa-443f-aedf-327ef9710f2a" containerName="istio-proxy"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664821    5898 cpu_manager.go:414] "containerMap: removing container" podUID="96f0bc26-fbf5-4d4c-a146-207dc12a4e20" containerName="istio-proxy"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664828    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96f0bc26-fbf5-4d4c-a146-207dc12a4e20" containerName="istio-proxy"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664840    5898 cpu_manager.go:414] "containerMap: removing container" podUID="70294a85-1050-4632-b3bf-c3e44a4c0b41" containerName="main"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664847    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70294a85-1050-4632-b3bf-c3e44a4c0b41" containerName="main"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664857    5898 cpu_manager.go:414] "containerMap: removing container" podUID="05eef4eb-f379-4dc7-a947-123bd85a4968" containerName="main"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664863    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05eef4eb-f379-4dc7-a947-123bd85a4968" containerName="main"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664874    5898 cpu_manager.go:414] "containerMap: removing container" podUID="fc35e0f4-fbc7-43e8-bf45-10d4d3ff19fe" containerName="payload-pre-processing"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664881    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc35e0f4-fbc7-43e8-bf45-10d4d3ff19fe" containerName="payload-pre-processing"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664894    5898 cpu_manager.go:414] "containerMap: removing container" podUID="b4dc6435-93bb-4b45-bdd5-3a80868e15e3" containerName="istio-proxy"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664932    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4dc6435-93bb-4b45-bdd5-3a80868e15e3" containerName="istio-proxy"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664948    5898 cpu_manager.go:414] "containerMap: removing container" podUID="3b386772-86df-4d1c-a5c4-fc27901eb9fd" containerName="istio-proxy"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664955    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b386772-86df-4d1c-a5c4-fc27901eb9fd" containerName="istio-proxy"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664964    5898 cpu_manager.go:414] "containerMap: removing container" podUID="aa609b95-6fbc-4cb6-9153-97e49232e302" containerName="istio-proxy"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664971    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa609b95-6fbc-4cb6-9153-97e49232e302" containerName="istio-proxy"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664983    5898 cpu_manager.go:414] "containerMap: removing container" podUID="b09804e3-8135-4f6d-ae34-c45d6da9728e" containerName="main"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.664990    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b09804e3-8135-4f6d-ae34-c45d6da9728e" containerName="main"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.665002    5898 cpu_manager.go:414] "containerMap: removing container" podUID="6f16bd6a-a7f0-49bb-844f-57b777bde2ed" containerName="main"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.665008    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f16bd6a-a7f0-49bb-844f-57b777bde2ed" containerName="main"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.665022    5898 cpu_manager.go:414] "containerMap: removing container" podUID="30460201-04af-49fb-a7d3-5ee121e3bdb4" containerName="payload-processing"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.665029    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30460201-04af-49fb-a7d3-5ee121e3bdb4" containerName="payload-processing"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.665039    5898 cpu_manager.go:414] "containerMap: removing container" podUID="c27f3b21-1447-4af9-8d99-c04fa9b23cda" containerName="main"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.665045    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c27f3b21-1447-4af9-8d99-c04fa9b23cda" containerName="main"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.665055    5898 cpu_manager.go:414] "containerMap: removing container" podUID="ced35944-e749-49e0-97ac-806b824c601c" containerName="istio-proxy"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.665062    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ced35944-e749-49e0-97ac-806b824c601c" containerName="istio-proxy"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.665072    5898 cpu_manager.go:414] "containerMap: removing container" podUID="1647f909-5580-4219-a2e1-30b4cd994379" containerName="main"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.665079    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1647f909-5580-4219-a2e1-30b4cd994379" containerName="main"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.665090    5898 cpu_manager.go:414] "containerMap: removing container" podUID="d4fd7194-084b-4339-9ea3-c5601858fed5" containerName="istio-proxy"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.665097    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4fd7194-084b-4339-9ea3-c5601858fed5" containerName="istio-proxy"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.665205    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="47ff272a-ac5c-4f59-bf7b-5c0692d09386" containerName="main"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.665218    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="268c3238-4e5e-4907-b5c0-cf47164701ea" containerName="istio-proxy"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.665230    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfd4e937-c236-4f33-8477-0c55aa536d30" containerName="maas-api"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.665239    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc5b5678-4d59-41b9-857b-8c502ec7eb28" containerName="payload-processing"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.665248    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="34fc87bb-0916-4cd3-bec8-d3ae0d7af7cd" containerName="istio-proxy"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.665261    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f19c225-09ec-43b9-af12-7c05b7546d72" containerName="payload-pre-processing"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.665272    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="b35f68e0-c492-44ba-a145-026a63713b60" containerName="maas-api"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.665282    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="3270ef0e-4775-48c6-8724-baff8e3fac06" containerName="payload-processing"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.665292    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d64c096-d3e4-4ef4-bc1a-08d5dfc97fff" containerName="maas-api"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.665302    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c2b56cf-6fbe-4787-86db-1692ecbb41e0" containerName="istio-proxy"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.665312    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa97913c-19e1-432c-9bd2-b1ce3278f7a5" containerName="istio-proxy"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.665322    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="d523008f-c740-4a41-89cb-e82e1ffa0094" containerName="main"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.665330    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="38428a53-04c7-400a-9c78-52407e982478" containerName="main"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.665339    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5620415-5516-441d-ab13-3a574d363b59" containerName="istio-proxy"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.665348    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8daa6d5-495f-4b98-abac-6dc2f9b4dbbc" containerName="maas-api"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.665356    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d3b67cf-5656-4d62-8adc-91df8cdcf394" containerName="maas-api"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.665364    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e630691-ed49-45ba-ac72-67ef295e80f3" containerName="istio-proxy"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.665373    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bc29954-1331-4bdf-9d6c-57e5894a5947" containerName="istio-proxy"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.665382    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="edceaaa4-f231-474e-85e1-b877ede202ef" containerName="istio-proxy"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.665391    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="c00385be-b173-422b-8ba6-0985c9cf7598" containerName="payload-pre-processing"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.665400    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfe8c234-c443-44a6-8420-9566768d6d02" containerName="istio-proxy"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.665410    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="59cf2966-7359-48ba-bd83-769f0c076397" containerName="istio-proxy"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.665417    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="d77a7956-ec67-4cfb-ba98-fd1e6ea6f227" containerName="istio-proxy"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.665425    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b3a9400-d8c6-4d4d-8be3-92cb66a4bb9f" containerName="istio-proxy"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.665435    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="7063ee78-5b1b-4f4a-b1f0-7898af800db1" containerName="payload-pre-processing"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.665445    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="fecce31e-a6fa-443f-aedf-327ef9710f2a" containerName="istio-proxy"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.665452    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="96f0bc26-fbf5-4d4c-a146-207dc12a4e20" containerName="istio-proxy"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.665462    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="70294a85-1050-4632-b3bf-c3e44a4c0b41" containerName="main"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.665469    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="05eef4eb-f379-4dc7-a947-123bd85a4968" containerName="main"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.665479    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc35e0f4-fbc7-43e8-bf45-10d4d3ff19fe" containerName="payload-pre-processing"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.665488    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4dc6435-93bb-4b45-bdd5-3a80868e15e3" containerName="istio-proxy"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.665498    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b386772-86df-4d1c-a5c4-fc27901eb9fd" containerName="istio-proxy"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.665506    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa609b95-6fbc-4cb6-9153-97e49232e302" containerName="istio-proxy"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.665513    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="b09804e3-8135-4f6d-ae34-c45d6da9728e" containerName="main"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.665523    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f16bd6a-a7f0-49bb-844f-57b777bde2ed" containerName="main"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.665533    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="30460201-04af-49fb-a7d3-5ee121e3bdb4" containerName="payload-processing"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.665541    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="c27f3b21-1447-4af9-8d99-c04fa9b23cda" containerName="main"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.665549    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="ced35944-e749-49e0-97ac-806b824c601c" containerName="istio-proxy"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.665558    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="1647f909-5580-4219-a2e1-30b4cd994379" containerName="main"
Oct 30 05:39:59.666916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.665567    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4fd7194-084b-4339-9ea3-c5601858fed5" containerName="istio-proxy"
Oct 30 05:39:59.670369 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.670350    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-157dec48-qpbkd"
Oct 30 05:39:59.688491 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.687356    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-157dec48-qpbkd"]
Oct 30 05:39:59.739997 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.739968    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlt67\" (UniqueName: \"kubernetes.io/projected/274129ce-5c11-4060-ab39-856dd097ffb1-kube-api-access-nlt67\") pod \"maas-api-revoke-sub-157dec48-qpbkd\" (UID: \"274129ce-5c11-4060-ab39-856dd097ffb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-157dec48-qpbkd"
Oct 30 05:39:59.740135 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.740072    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/274129ce-5c11-4060-ab39-856dd097ffb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-157dec48-qpbkd\" (UID: \"274129ce-5c11-4060-ab39-856dd097ffb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-157dec48-qpbkd"
Oct 30 05:39:59.740176 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.740146    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/274129ce-5c11-4060-ab39-856dd097ffb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-157dec48-qpbkd\" (UID: \"274129ce-5c11-4060-ab39-856dd097ffb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-157dec48-qpbkd"
Oct 30 05:39:59.780231 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.780207    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:39:59.780231 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.780218    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 05:39:59.780409 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.780267    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\""
Oct 30 05:39:59.780409 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.780279    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 05:39:59.840944 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.840889    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/274129ce-5c11-4060-ab39-856dd097ffb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-157dec48-qpbkd\" (UID: \"274129ce-5c11-4060-ab39-856dd097ffb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-157dec48-qpbkd"
Oct 30 05:39:59.841068 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.841003    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/274129ce-5c11-4060-ab39-856dd097ffb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-157dec48-qpbkd\" (UID: \"274129ce-5c11-4060-ab39-856dd097ffb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-157dec48-qpbkd"
Oct 30 05:39:59.841068 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.841041    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlt67\" (UniqueName: \"kubernetes.io/projected/274129ce-5c11-4060-ab39-856dd097ffb1-kube-api-access-nlt67\") pod \"maas-api-revoke-sub-157dec48-qpbkd\" (UID: \"274129ce-5c11-4060-ab39-856dd097ffb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-157dec48-qpbkd"
Oct 30 05:39:59.841533 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.841517    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/274129ce-5c11-4060-ab39-856dd097ffb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-157dec48-qpbkd\" (UID: \"274129ce-5c11-4060-ab39-856dd097ffb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-157dec48-qpbkd"
Oct 30 05:39:59.843367 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.843351    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/274129ce-5c11-4060-ab39-856dd097ffb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-157dec48-qpbkd\" (UID: \"274129ce-5c11-4060-ab39-856dd097ffb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-157dec48-qpbkd"
Oct 30 05:39:59.860966 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.860947    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlt67\" (UniqueName: \"kubernetes.io/projected/274129ce-5c11-4060-ab39-856dd097ffb1-kube-api-access-nlt67\") pod \"maas-api-revoke-sub-157dec48-qpbkd\" (UID: \"274129ce-5c11-4060-ab39-856dd097ffb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-157dec48-qpbkd"
Oct 30 05:39:59.983594 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:39:59.983520    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-157dec48-qpbkd"
Oct 30 05:40:00.000913 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:00.000869    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:40:00.304651 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:00.304583    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-157dec48-qpbkd"]
Oct 30 05:40:00.308337 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:40:00.308308    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod274129ce_5c11_4060_ab39_856dd097ffb1.slice/crio-09acc635ab9d1e0a175f91017c8b2b28fe5e404244b1fbcfbb6b01784a83f725 WatchSource:0}: Error finding container 09acc635ab9d1e0a175f91017c8b2b28fe5e404244b1fbcfbb6b01784a83f725: Status 404 returned error can't find the container with id 09acc635ab9d1e0a175f91017c8b2b28fe5e404244b1fbcfbb6b01784a83f725
Oct 30 05:40:00.542548 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:00.542517    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-157dec48-qpbkd" event={"ID":"274129ce-5c11-4060-ab39-856dd097ffb1","Type":"ContainerStarted","Data":"09acc635ab9d1e0a175f91017c8b2b28fe5e404244b1fbcfbb6b01784a83f725"}
Oct 30 05:40:01.525034 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:01.524938    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:40:01.552391 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:01.552357    5898 generic.go:356] "Generic (PLEG): container finished" podID="274129ce-5c11-4060-ab39-856dd097ffb1" containerID="24bede84efbc610521ee26d72b14c7da371b048cf8c5e17278439fdca0377154" exitCode=0
Oct 30 05:40:01.552575 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:01.552446    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-157dec48-qpbkd" event={"ID":"274129ce-5c11-4060-ab39-856dd097ffb1","Type":"ContainerDied","Data":"24bede84efbc610521ee26d72b14c7da371b048cf8c5e17278439fdca0377154"}
Oct 30 05:40:02.126991 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:02.126946    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:40:02.247087 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:02.247049    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:40:02.366886 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:02.366856    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:40:02.702991 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:02.702967    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-157dec48-qpbkd"
Oct 30 05:40:02.773120 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:02.773086    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/274129ce-5c11-4060-ab39-856dd097ffb1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/274129ce-5c11-4060-ab39-856dd097ffb1-maas-api-xks-ca\") pod \"274129ce-5c11-4060-ab39-856dd097ffb1\" (UID: \"274129ce-5c11-4060-ab39-856dd097ffb1\") "
Oct 30 05:40:02.773299 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:02.773149    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/274129ce-5c11-4060-ab39-856dd097ffb1-kube-api-access-nlt67\" (UniqueName: \"kubernetes.io/projected/274129ce-5c11-4060-ab39-856dd097ffb1-kube-api-access-nlt67\") pod \"274129ce-5c11-4060-ab39-856dd097ffb1\" (UID: \"274129ce-5c11-4060-ab39-856dd097ffb1\") "
Oct 30 05:40:02.773299 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:02.773252    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/274129ce-5c11-4060-ab39-856dd097ffb1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/274129ce-5c11-4060-ab39-856dd097ffb1-maas-api-openshift-ca\") pod \"274129ce-5c11-4060-ab39-856dd097ffb1\" (UID: \"274129ce-5c11-4060-ab39-856dd097ffb1\") "
Oct 30 05:40:02.773620 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:02.773580    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/274129ce-5c11-4060-ab39-856dd097ffb1-maas-api-openshift-ca" pod "274129ce-5c11-4060-ab39-856dd097ffb1" (UID: "274129ce-5c11-4060-ab39-856dd097ffb1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:40:02.775342 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:02.775309    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/274129ce-5c11-4060-ab39-856dd097ffb1-maas-api-xks-ca" pod "274129ce-5c11-4060-ab39-856dd097ffb1" (UID: "274129ce-5c11-4060-ab39-856dd097ffb1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:40:02.775465 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:02.775433    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/274129ce-5c11-4060-ab39-856dd097ffb1-kube-api-access-nlt67" pod "274129ce-5c11-4060-ab39-856dd097ffb1" (UID: "274129ce-5c11-4060-ab39-856dd097ffb1"). InnerVolumeSpecName "kube-api-access-nlt67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:40:02.874316 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:02.874283    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/274129ce-5c11-4060-ab39-856dd097ffb1-maas-api-xks-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:02.874316 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:02.874313    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlt67\" (UniqueName: \"kubernetes.io/projected/274129ce-5c11-4060-ab39-856dd097ffb1-kube-api-access-nlt67\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:02.874508 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:02.874328    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/274129ce-5c11-4060-ab39-856dd097ffb1-maas-api-openshift-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:02.924399 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:02.924367    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:40:03.340110 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.340075    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-0b21ba/distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6" podUID="d30a7a86-f1da-451c-81dd-ebdbaeb4f8af" containerName="main" containerID="cri-o://3e50dbcf9a7bdcde2a0c56f382bad3a52e565f5ffed875946cf4f38c2755d4cd" gracePeriod=45
Oct 30 05:40:03.351337 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.351305    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-0b21ba/distinct-w5-0b21ba-kserve-7674c59666-m2sbd" podUID="24c6b2d4-3377-4079-8273-3f48064c3a98" containerName="main" containerID="cri-o://762151b43e59b658a751c9ded7fc81204ed50a26626d7edea28be74b3b45d23f" gracePeriod=45
Oct 30 05:40:03.353460 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.353355    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849" podUID="a9707f50-d5e1-409f-b8f3-c58629677aaf" containerName="main" containerID="cri-o://7847063f47a43c40439f8270962e51a01a8a90739899dc81fcdc3bf48373768a" gracePeriod=45
Oct 30 05:40:03.564976 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.564943    5898 generic.go:356] "Generic (PLEG): container finished" podID="d30a7a86-f1da-451c-81dd-ebdbaeb4f8af" containerID="3e50dbcf9a7bdcde2a0c56f382bad3a52e565f5ffed875946cf4f38c2755d4cd" exitCode=0
Oct 30 05:40:03.565157 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.565012    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-0b21ba/distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6" event={"ID":"d30a7a86-f1da-451c-81dd-ebdbaeb4f8af","Type":"ContainerDied","Data":"3e50dbcf9a7bdcde2a0c56f382bad3a52e565f5ffed875946cf4f38c2755d4cd"}
Oct 30 05:40:03.566493 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.566465    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-157dec48-qpbkd" event={"ID":"274129ce-5c11-4060-ab39-856dd097ffb1","Type":"ContainerDied","Data":"09acc635ab9d1e0a175f91017c8b2b28fe5e404244b1fbcfbb6b01784a83f725"}
Oct 30 05:40:03.566640 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.566510    5898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09acc635ab9d1e0a175f91017c8b2b28fe5e404244b1fbcfbb6b01784a83f725"
Oct 30 05:40:03.566640 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.566478    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-157dec48-qpbkd"
Oct 30 05:40:03.568417 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.568383    5898 generic.go:356] "Generic (PLEG): container finished" podID="24c6b2d4-3377-4079-8273-3f48064c3a98" containerID="762151b43e59b658a751c9ded7fc81204ed50a26626d7edea28be74b3b45d23f" exitCode=0
Oct 30 05:40:03.568535 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.568460    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-0b21ba/distinct-w5-0b21ba-kserve-7674c59666-m2sbd" event={"ID":"24c6b2d4-3377-4079-8273-3f48064c3a98","Type":"ContainerDied","Data":"762151b43e59b658a751c9ded7fc81204ed50a26626d7edea28be74b3b45d23f"}
Oct 30 05:40:03.570816 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.570794    5898 generic.go:356] "Generic (PLEG): container finished" podID="a9707f50-d5e1-409f-b8f3-c58629677aaf" containerID="7847063f47a43c40439f8270962e51a01a8a90739899dc81fcdc3bf48373768a" exitCode=0
Oct 30 05:40:03.570959 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.570833    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849" event={"ID":"a9707f50-d5e1-409f-b8f3-c58629677aaf","Type":"ContainerDied","Data":"7847063f47a43c40439f8270962e51a01a8a90739899dc81fcdc3bf48373768a"}
Oct 30 05:40:03.600422 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.600396    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849"
Oct 30 05:40:03.637070 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.637046    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-0b21ba/distinct-w5-0b21ba-kserve-7674c59666-m2sbd"
Oct 30 05:40:03.650581 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.650560    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-0b21ba/distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6"
Oct 30 05:40:03.684435 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.684395    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9707f50-d5e1-409f-b8f3-c58629677aaf-home\" (UniqueName: \"kubernetes.io/empty-dir/a9707f50-d5e1-409f-b8f3-c58629677aaf-home\") pod \"a9707f50-d5e1-409f-b8f3-c58629677aaf\" (UID: \"a9707f50-d5e1-409f-b8f3-c58629677aaf\") "
Oct 30 05:40:03.684624 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.684455    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24c6b2d4-3377-4079-8273-3f48064c3a98-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24c6b2d4-3377-4079-8273-3f48064c3a98-model-cache\") pod \"24c6b2d4-3377-4079-8273-3f48064c3a98\" (UID: \"24c6b2d4-3377-4079-8273-3f48064c3a98\") "
Oct 30 05:40:03.684624 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.684487    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24c6b2d4-3377-4079-8273-3f48064c3a98-tls-certs\" (UniqueName: \"kubernetes.io/secret/24c6b2d4-3377-4079-8273-3f48064c3a98-tls-certs\") pod \"24c6b2d4-3377-4079-8273-3f48064c3a98\" (UID: \"24c6b2d4-3377-4079-8273-3f48064c3a98\") "
Oct 30 05:40:03.684624 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.684515    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24c6b2d4-3377-4079-8273-3f48064c3a98-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24c6b2d4-3377-4079-8273-3f48064c3a98-tmp-dir\") pod \"24c6b2d4-3377-4079-8273-3f48064c3a98\" (UID: \"24c6b2d4-3377-4079-8273-3f48064c3a98\") "
Oct 30 05:40:03.684624 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.684550    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9707f50-d5e1-409f-b8f3-c58629677aaf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9707f50-d5e1-409f-b8f3-c58629677aaf-tmp-dir\") pod \"a9707f50-d5e1-409f-b8f3-c58629677aaf\" (UID: \"a9707f50-d5e1-409f-b8f3-c58629677aaf\") "
Oct 30 05:40:03.684624 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.684584    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d30a7a86-f1da-451c-81dd-ebdbaeb4f8af-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d30a7a86-f1da-451c-81dd-ebdbaeb4f8af-model-cache\") pod \"d30a7a86-f1da-451c-81dd-ebdbaeb4f8af\" (UID: \"d30a7a86-f1da-451c-81dd-ebdbaeb4f8af\") "
Oct 30 05:40:03.684963 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.684645    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9707f50-d5e1-409f-b8f3-c58629677aaf-kube-api-access-jvbr9\" (UniqueName: \"kubernetes.io/projected/a9707f50-d5e1-409f-b8f3-c58629677aaf-kube-api-access-jvbr9\") pod \"a9707f50-d5e1-409f-b8f3-c58629677aaf\" (UID: \"a9707f50-d5e1-409f-b8f3-c58629677aaf\") "
Oct 30 05:40:03.684963 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.684692    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24c6b2d4-3377-4079-8273-3f48064c3a98-dshm\" (UniqueName: \"kubernetes.io/empty-dir/24c6b2d4-3377-4079-8273-3f48064c3a98-dshm\") pod \"24c6b2d4-3377-4079-8273-3f48064c3a98\" (UID: \"24c6b2d4-3377-4079-8273-3f48064c3a98\") "
Oct 30 05:40:03.684963 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.684742    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24c6b2d4-3377-4079-8273-3f48064c3a98-home\" (UniqueName: \"kubernetes.io/empty-dir/24c6b2d4-3377-4079-8273-3f48064c3a98-home\") pod \"24c6b2d4-3377-4079-8273-3f48064c3a98\" (UID: \"24c6b2d4-3377-4079-8273-3f48064c3a98\") "
Oct 30 05:40:03.684963 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.684790    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d30a7a86-f1da-451c-81dd-ebdbaeb4f8af-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d30a7a86-f1da-451c-81dd-ebdbaeb4f8af-dshm\") pod \"d30a7a86-f1da-451c-81dd-ebdbaeb4f8af\" (UID: \"d30a7a86-f1da-451c-81dd-ebdbaeb4f8af\") "
Oct 30 05:40:03.684963 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.684822    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d30a7a86-f1da-451c-81dd-ebdbaeb4f8af-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d30a7a86-f1da-451c-81dd-ebdbaeb4f8af-tmp-dir\") pod \"d30a7a86-f1da-451c-81dd-ebdbaeb4f8af\" (UID: \"d30a7a86-f1da-451c-81dd-ebdbaeb4f8af\") "
Oct 30 05:40:03.684963 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.684865    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d30a7a86-f1da-451c-81dd-ebdbaeb4f8af-home\" (UniqueName: \"kubernetes.io/empty-dir/d30a7a86-f1da-451c-81dd-ebdbaeb4f8af-home\") pod \"d30a7a86-f1da-451c-81dd-ebdbaeb4f8af\" (UID: \"d30a7a86-f1da-451c-81dd-ebdbaeb4f8af\") "
Oct 30 05:40:03.684963 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.684951    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d30a7a86-f1da-451c-81dd-ebdbaeb4f8af-kube-api-access-s2qrr\" (UniqueName: \"kubernetes.io/projected/d30a7a86-f1da-451c-81dd-ebdbaeb4f8af-kube-api-access-s2qrr\") pod \"d30a7a86-f1da-451c-81dd-ebdbaeb4f8af\" (UID: \"d30a7a86-f1da-451c-81dd-ebdbaeb4f8af\") "
Oct 30 05:40:03.685345 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.684992    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9707f50-d5e1-409f-b8f3-c58629677aaf-tls-certs\" (UniqueName: \"kubernetes.io/secret/a9707f50-d5e1-409f-b8f3-c58629677aaf-tls-certs\") pod \"a9707f50-d5e1-409f-b8f3-c58629677aaf\" (UID: \"a9707f50-d5e1-409f-b8f3-c58629677aaf\") "
Oct 30 05:40:03.685345 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.685029    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24c6b2d4-3377-4079-8273-3f48064c3a98-kube-api-access-8cblr\" (UniqueName: \"kubernetes.io/projected/24c6b2d4-3377-4079-8273-3f48064c3a98-kube-api-access-8cblr\") pod \"24c6b2d4-3377-4079-8273-3f48064c3a98\" (UID: \"24c6b2d4-3377-4079-8273-3f48064c3a98\") "
Oct 30 05:40:03.685345 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.685095    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9707f50-d5e1-409f-b8f3-c58629677aaf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9707f50-d5e1-409f-b8f3-c58629677aaf-dshm\") pod \"a9707f50-d5e1-409f-b8f3-c58629677aaf\" (UID: \"a9707f50-d5e1-409f-b8f3-c58629677aaf\") "
Oct 30 05:40:03.685345 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.685134    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d30a7a86-f1da-451c-81dd-ebdbaeb4f8af-tls-certs\" (UniqueName: \"kubernetes.io/secret/d30a7a86-f1da-451c-81dd-ebdbaeb4f8af-tls-certs\") pod \"d30a7a86-f1da-451c-81dd-ebdbaeb4f8af\" (UID: \"d30a7a86-f1da-451c-81dd-ebdbaeb4f8af\") "
Oct 30 05:40:03.685345 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.685164    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9707f50-d5e1-409f-b8f3-c58629677aaf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9707f50-d5e1-409f-b8f3-c58629677aaf-model-cache\") pod \"a9707f50-d5e1-409f-b8f3-c58629677aaf\" (UID: \"a9707f50-d5e1-409f-b8f3-c58629677aaf\") "
Oct 30 05:40:03.685963 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.685895    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9707f50-d5e1-409f-b8f3-c58629677aaf-model-cache" pod "a9707f50-d5e1-409f-b8f3-c58629677aaf" (UID: "a9707f50-d5e1-409f-b8f3-c58629677aaf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:40:03.686147 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.686077    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24c6b2d4-3377-4079-8273-3f48064c3a98-model-cache" pod "24c6b2d4-3377-4079-8273-3f48064c3a98" (UID: "24c6b2d4-3377-4079-8273-3f48064c3a98"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:40:03.686483 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.686271    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9707f50-d5e1-409f-b8f3-c58629677aaf-home" pod "a9707f50-d5e1-409f-b8f3-c58629677aaf" (UID: "a9707f50-d5e1-409f-b8f3-c58629677aaf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:40:03.686582 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.686567    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24c6b2d4-3377-4079-8273-3f48064c3a98-tmp-dir" pod "24c6b2d4-3377-4079-8273-3f48064c3a98" (UID: "24c6b2d4-3377-4079-8273-3f48064c3a98"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:40:03.686669 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.685955    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24c6b2d4-3377-4079-8273-3f48064c3a98-home" pod "24c6b2d4-3377-4079-8273-3f48064c3a98" (UID: "24c6b2d4-3377-4079-8273-3f48064c3a98"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:40:03.686998 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.686977    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9707f50-d5e1-409f-b8f3-c58629677aaf-tmp-dir" pod "a9707f50-d5e1-409f-b8f3-c58629677aaf" (UID: "a9707f50-d5e1-409f-b8f3-c58629677aaf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:40:03.687181 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.687158    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d30a7a86-f1da-451c-81dd-ebdbaeb4f8af-home" pod "d30a7a86-f1da-451c-81dd-ebdbaeb4f8af" (UID: "d30a7a86-f1da-451c-81dd-ebdbaeb4f8af"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:40:03.688270 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.688248    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d30a7a86-f1da-451c-81dd-ebdbaeb4f8af-model-cache" pod "d30a7a86-f1da-451c-81dd-ebdbaeb4f8af" (UID: "d30a7a86-f1da-451c-81dd-ebdbaeb4f8af"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:40:03.689344 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.689232    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d30a7a86-f1da-451c-81dd-ebdbaeb4f8af-tmp-dir" pod "d30a7a86-f1da-451c-81dd-ebdbaeb4f8af" (UID: "d30a7a86-f1da-451c-81dd-ebdbaeb4f8af"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:40:03.693821 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.693796    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9707f50-d5e1-409f-b8f3-c58629677aaf-tls-certs" pod "a9707f50-d5e1-409f-b8f3-c58629677aaf" (UID: "a9707f50-d5e1-409f-b8f3-c58629677aaf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:40:03.694840 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.694814    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24c6b2d4-3377-4079-8273-3f48064c3a98-dshm" pod "24c6b2d4-3377-4079-8273-3f48064c3a98" (UID: "24c6b2d4-3377-4079-8273-3f48064c3a98"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:40:03.695017 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.694877    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d30a7a86-f1da-451c-81dd-ebdbaeb4f8af-kube-api-access-s2qrr" pod "d30a7a86-f1da-451c-81dd-ebdbaeb4f8af" (UID: "d30a7a86-f1da-451c-81dd-ebdbaeb4f8af"). InnerVolumeSpecName "kube-api-access-s2qrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:40:03.695156 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.695134    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d30a7a86-f1da-451c-81dd-ebdbaeb4f8af-tls-certs" pod "d30a7a86-f1da-451c-81dd-ebdbaeb4f8af" (UID: "d30a7a86-f1da-451c-81dd-ebdbaeb4f8af"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:40:03.695231 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.695212    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9707f50-d5e1-409f-b8f3-c58629677aaf-dshm" pod "a9707f50-d5e1-409f-b8f3-c58629677aaf" (UID: "a9707f50-d5e1-409f-b8f3-c58629677aaf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:40:03.695292 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.695240    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d30a7a86-f1da-451c-81dd-ebdbaeb4f8af-dshm" pod "d30a7a86-f1da-451c-81dd-ebdbaeb4f8af" (UID: "d30a7a86-f1da-451c-81dd-ebdbaeb4f8af"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:40:03.696227 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.696097    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24c6b2d4-3377-4079-8273-3f48064c3a98-tls-certs" pod "24c6b2d4-3377-4079-8273-3f48064c3a98" (UID: "24c6b2d4-3377-4079-8273-3f48064c3a98"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:40:03.696332 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.696241    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9707f50-d5e1-409f-b8f3-c58629677aaf-kube-api-access-jvbr9" pod "a9707f50-d5e1-409f-b8f3-c58629677aaf" (UID: "a9707f50-d5e1-409f-b8f3-c58629677aaf"). InnerVolumeSpecName "kube-api-access-jvbr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:40:03.696410 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.696390    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24c6b2d4-3377-4079-8273-3f48064c3a98-kube-api-access-8cblr" pod "24c6b2d4-3377-4079-8273-3f48064c3a98" (UID: "24c6b2d4-3377-4079-8273-3f48064c3a98"). InnerVolumeSpecName "kube-api-access-8cblr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:40:03.787256 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.787211    5898 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24c6b2d4-3377-4079-8273-3f48064c3a98-home\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:03.787256 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.787247    5898 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d30a7a86-f1da-451c-81dd-ebdbaeb4f8af-dshm\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:03.787256 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.787260    5898 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d30a7a86-f1da-451c-81dd-ebdbaeb4f8af-tmp-dir\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:03.787895 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.787273    5898 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d30a7a86-f1da-451c-81dd-ebdbaeb4f8af-home\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:03.787895 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.787286    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2qrr\" (UniqueName: \"kubernetes.io/projected/d30a7a86-f1da-451c-81dd-ebdbaeb4f8af-kube-api-access-s2qrr\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:03.787895 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.787301    5898 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9707f50-d5e1-409f-b8f3-c58629677aaf-tls-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:03.787895 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.787316    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cblr\" (UniqueName: \"kubernetes.io/projected/24c6b2d4-3377-4079-8273-3f48064c3a98-kube-api-access-8cblr\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:03.787895 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.787328    5898 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9707f50-d5e1-409f-b8f3-c58629677aaf-dshm\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:03.787895 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.787341    5898 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d30a7a86-f1da-451c-81dd-ebdbaeb4f8af-tls-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:03.787895 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.787354    5898 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9707f50-d5e1-409f-b8f3-c58629677aaf-model-cache\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:03.787895 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.787366    5898 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9707f50-d5e1-409f-b8f3-c58629677aaf-home\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:03.787895 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.787377    5898 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24c6b2d4-3377-4079-8273-3f48064c3a98-model-cache\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:03.787895 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.787390    5898 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24c6b2d4-3377-4079-8273-3f48064c3a98-tls-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:03.787895 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.787404    5898 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24c6b2d4-3377-4079-8273-3f48064c3a98-tmp-dir\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:03.787895 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.787416    5898 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9707f50-d5e1-409f-b8f3-c58629677aaf-tmp-dir\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:03.787895 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.787428    5898 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d30a7a86-f1da-451c-81dd-ebdbaeb4f8af-model-cache\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:03.787895 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.787441    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvbr9\" (UniqueName: \"kubernetes.io/projected/a9707f50-d5e1-409f-b8f3-c58629677aaf-kube-api-access-jvbr9\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:03.787895 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:03.787453    5898 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24c6b2d4-3377-4079-8273-3f48064c3a98-dshm\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:04.309997 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:04.309965    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:40:04.575069 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:04.575027    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0b21ba/distinct-w5-0b21ba-kserve-7674c59666-m2sbd" podUID="24c6b2d4-3377-4079-8273-3f48064c3a98" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:40:04.578130 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:04.578103    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-0b21ba/distinct-w5-0b21ba-kserve-7674c59666-m2sbd"
Oct 30 05:40:04.578307 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:04.578103    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-0b21ba/distinct-w5-0b21ba-kserve-7674c59666-m2sbd" event={"ID":"24c6b2d4-3377-4079-8273-3f48064c3a98","Type":"ContainerDied","Data":"47bc0f16f05b93bd8c57a366b89d2baf0be82577fe82cd9fcb3db0b3517142c4"}
Oct 30 05:40:04.578307 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:04.578241    5898 scope.go:122] "RemoveContainer" containerID="762151b43e59b658a751c9ded7fc81204ed50a26626d7edea28be74b3b45d23f"
Oct 30 05:40:04.579589 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:04.579571    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849"
Oct 30 05:40:04.579735 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:04.579624    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849" event={"ID":"a9707f50-d5e1-409f-b8f3-c58629677aaf","Type":"ContainerDied","Data":"5e746f6592babe76365d865b5926390fc3734cbd453092ddb115296418c46392"}
Oct 30 05:40:04.581310 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:04.581288    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-0b21ba/distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6" event={"ID":"d30a7a86-f1da-451c-81dd-ebdbaeb4f8af","Type":"ContainerDied","Data":"cf0111a03ec810d6f13daec0155e02b31358852d892538ca4b8bd24a9959e6b1"}
Oct 30 05:40:04.581448 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:04.581295    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-0b21ba/distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6"
Oct 30 05:40:04.589504 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:04.589484    5898 scope.go:122] "RemoveContainer" containerID="7847063f47a43c40439f8270962e51a01a8a90739899dc81fcdc3bf48373768a"
Oct 30 05:40:04.598862 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:04.597847    5898 scope.go:122] "RemoveContainer" containerID="3e50dbcf9a7bdcde2a0c56f382bad3a52e565f5ffed875946cf4f38c2755d4cd"
Oct 30 05:40:04.629022 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:04.628998    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849"]
Oct 30 05:40:04.639996 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:04.639974    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-0b21ba/premium-w5-0b21ba-kserve-c8888d867-9z849"]
Oct 30 05:40:04.667327 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:04.665611    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-0b21ba/distinct-w5-0b21ba-kserve-7674c59666-m2sbd"]
Oct 30 05:40:04.670309 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:04.670287    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-0b21ba/distinct-w5-0b21ba-kserve-7674c59666-m2sbd"]
Oct 30 05:40:04.708126 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:04.708103    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-0b21ba/distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6"]
Oct 30 05:40:04.713760 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:04.713740    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-0b21ba/distinct-2-w5-0b21ba-kserve-64d78cdb57-k5cl6"]
Oct 30 05:40:04.888524 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:04.888453    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24c6b2d4-3377-4079-8273-3f48064c3a98" path="/var/lib/kubelet/pods/24c6b2d4-3377-4079-8273-3f48064c3a98/volumes"
Oct 30 05:40:04.888881 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:04.888867    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9707f50-d5e1-409f-b8f3-c58629677aaf" path="/var/lib/kubelet/pods/a9707f50-d5e1-409f-b8f3-c58629677aaf/volumes"
Oct 30 05:40:04.889271 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:04.889257    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d30a7a86-f1da-451c-81dd-ebdbaeb4f8af" path="/var/lib/kubelet/pods/d30a7a86-f1da-451c-81dd-ebdbaeb4f8af/volumes"
Oct 30 05:40:05.795552 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:05.795499    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:40:06.125413 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:06.125382    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:40:07.964200 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:07.964160    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:40:09.368070 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:09.368039    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:40:09.536291 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:09.536257    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:40:09.817226 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:09.817149    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54"]
Oct 30 05:40:09.817438 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:09.817410    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54" podUID="033d7e0d-2790-46af-829c-78374de43cd1" containerName="istio-proxy" containerID="cri-o://2143370d428514c87be5c439b81b6fcd172b9051c43146d903a849096688c2f7" gracePeriod=30
Oct 30 05:40:09.827618 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:09.827591    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7"]
Oct 30 05:40:09.827826 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:09.827805    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7" podUID="b39e7705-286b-4d9a-8d3a-41dbfa9e20f2" containerName="istio-proxy" containerID="cri-o://62cce9c2945e944d0b4851cc15b0904e970c4a8ee45585b94354f5f6b026113c" gracePeriod=30
Oct 30 05:40:09.840095 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:09.840072    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:40:09.951979 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:09.951945    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:40:10.371623 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:10.371584    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:40:11.743408 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:11.743375    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:40:12.299857 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:12.299823    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 05:40:12.300053 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:12.299877    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54" podUID="033d7e0d-2790-46af-829c-78374de43cd1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 05:40:12.320317 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:12.320288    5898 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 05:40:12.320467 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:12.320342    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7" podUID="b39e7705-286b-4d9a-8d3a-41dbfa9e20f2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 05:40:13.323592 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:13.323549    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:40:13.838943 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:13.838861    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:40:13.988137 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:13.988100    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:40:14.101214 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:14.101137    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:40:14.565413 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:14.565324    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:40:14.566051 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:14.566025    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:40:15.078671 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.078647    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54"
Oct 30 05:40:15.124991 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.124962    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7"
Oct 30 05:40:15.196839 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.196761    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/033d7e0d-2790-46af-829c-78374de43cd1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/033d7e0d-2790-46af-829c-78374de43cd1-istio-envoy\") pod \"033d7e0d-2790-46af-829c-78374de43cd1\" (UID: \"033d7e0d-2790-46af-829c-78374de43cd1\") "
Oct 30 05:40:15.197168 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.197141    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b39e7705-286b-4d9a-8d3a-41dbfa9e20f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b39e7705-286b-4d9a-8d3a-41dbfa9e20f2-istio-envoy\") pod \"b39e7705-286b-4d9a-8d3a-41dbfa9e20f2\" (UID: \"b39e7705-286b-4d9a-8d3a-41dbfa9e20f2\") "
Oct 30 05:40:15.197382 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.197184    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/033d7e0d-2790-46af-829c-78374de43cd1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/033d7e0d-2790-46af-829c-78374de43cd1-istio-data\") pod \"033d7e0d-2790-46af-829c-78374de43cd1\" (UID: \"033d7e0d-2790-46af-829c-78374de43cd1\") "
Oct 30 05:40:15.197382 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.197229    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b39e7705-286b-4d9a-8d3a-41dbfa9e20f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b39e7705-286b-4d9a-8d3a-41dbfa9e20f2-istiod-ca-cert\") pod \"b39e7705-286b-4d9a-8d3a-41dbfa9e20f2\" (UID: \"b39e7705-286b-4d9a-8d3a-41dbfa9e20f2\") "
Oct 30 05:40:15.197382 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.197273    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/033d7e0d-2790-46af-829c-78374de43cd1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/033d7e0d-2790-46af-829c-78374de43cd1-workload-certs\") pod \"033d7e0d-2790-46af-829c-78374de43cd1\" (UID: \"033d7e0d-2790-46af-829c-78374de43cd1\") "
Oct 30 05:40:15.197382 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.197337    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b39e7705-286b-4d9a-8d3a-41dbfa9e20f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39e7705-286b-4d9a-8d3a-41dbfa9e20f2-workload-socket\") pod \"b39e7705-286b-4d9a-8d3a-41dbfa9e20f2\" (UID: \"b39e7705-286b-4d9a-8d3a-41dbfa9e20f2\") "
Oct 30 05:40:15.197382 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.197374    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/033d7e0d-2790-46af-829c-78374de43cd1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/033d7e0d-2790-46af-829c-78374de43cd1-istio-podinfo\") pod \"033d7e0d-2790-46af-829c-78374de43cd1\" (UID: \"033d7e0d-2790-46af-829c-78374de43cd1\") "
Oct 30 05:40:15.197692 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.197405    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b39e7705-286b-4d9a-8d3a-41dbfa9e20f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39e7705-286b-4d9a-8d3a-41dbfa9e20f2-credential-socket\") pod \"b39e7705-286b-4d9a-8d3a-41dbfa9e20f2\" (UID: \"b39e7705-286b-4d9a-8d3a-41dbfa9e20f2\") "
Oct 30 05:40:15.197692 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.197433    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b39e7705-286b-4d9a-8d3a-41dbfa9e20f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b39e7705-286b-4d9a-8d3a-41dbfa9e20f2-istio-data\") pod \"b39e7705-286b-4d9a-8d3a-41dbfa9e20f2\" (UID: \"b39e7705-286b-4d9a-8d3a-41dbfa9e20f2\") "
Oct 30 05:40:15.197692 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.197466    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b39e7705-286b-4d9a-8d3a-41dbfa9e20f2-kube-api-access-k4pzq\" (UniqueName: \"kubernetes.io/projected/b39e7705-286b-4d9a-8d3a-41dbfa9e20f2-kube-api-access-k4pzq\") pod \"b39e7705-286b-4d9a-8d3a-41dbfa9e20f2\" (UID: \"b39e7705-286b-4d9a-8d3a-41dbfa9e20f2\") "
Oct 30 05:40:15.197692 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.197514    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b39e7705-286b-4d9a-8d3a-41dbfa9e20f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b39e7705-286b-4d9a-8d3a-41dbfa9e20f2-istio-podinfo\") pod \"b39e7705-286b-4d9a-8d3a-41dbfa9e20f2\" (UID: \"b39e7705-286b-4d9a-8d3a-41dbfa9e20f2\") "
Oct 30 05:40:15.197692 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.197542    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/033d7e0d-2790-46af-829c-78374de43cd1-istio-token\" (UniqueName: \"kubernetes.io/projected/033d7e0d-2790-46af-829c-78374de43cd1-istio-token\") pod \"033d7e0d-2790-46af-829c-78374de43cd1\" (UID: \"033d7e0d-2790-46af-829c-78374de43cd1\") "
Oct 30 05:40:15.197692 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.197579    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b39e7705-286b-4d9a-8d3a-41dbfa9e20f2-istio-token\" (UniqueName: \"kubernetes.io/projected/b39e7705-286b-4d9a-8d3a-41dbfa9e20f2-istio-token\") pod \"b39e7705-286b-4d9a-8d3a-41dbfa9e20f2\" (UID: \"b39e7705-286b-4d9a-8d3a-41dbfa9e20f2\") "
Oct 30 05:40:15.197692 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.197610    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/033d7e0d-2790-46af-829c-78374de43cd1-kube-api-access-hs2rh\" (UniqueName: \"kubernetes.io/projected/033d7e0d-2790-46af-829c-78374de43cd1-kube-api-access-hs2rh\") pod \"033d7e0d-2790-46af-829c-78374de43cd1\" (UID: \"033d7e0d-2790-46af-829c-78374de43cd1\") "
Oct 30 05:40:15.197692 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.197639    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/033d7e0d-2790-46af-829c-78374de43cd1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/033d7e0d-2790-46af-829c-78374de43cd1-credential-socket\") pod \"033d7e0d-2790-46af-829c-78374de43cd1\" (UID: \"033d7e0d-2790-46af-829c-78374de43cd1\") "
Oct 30 05:40:15.197692 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.197689    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b39e7705-286b-4d9a-8d3a-41dbfa9e20f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b39e7705-286b-4d9a-8d3a-41dbfa9e20f2-workload-certs\") pod \"b39e7705-286b-4d9a-8d3a-41dbfa9e20f2\" (UID: \"b39e7705-286b-4d9a-8d3a-41dbfa9e20f2\") "
Oct 30 05:40:15.199082 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.197718    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/033d7e0d-2790-46af-829c-78374de43cd1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/033d7e0d-2790-46af-829c-78374de43cd1-workload-socket\") pod \"033d7e0d-2790-46af-829c-78374de43cd1\" (UID: \"033d7e0d-2790-46af-829c-78374de43cd1\") "
Oct 30 05:40:15.199082 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.197794    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/033d7e0d-2790-46af-829c-78374de43cd1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/033d7e0d-2790-46af-829c-78374de43cd1-istiod-ca-cert\") pod \"033d7e0d-2790-46af-829c-78374de43cd1\" (UID: \"033d7e0d-2790-46af-829c-78374de43cd1\") "
Oct 30 05:40:15.199082 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.197980    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b39e7705-286b-4d9a-8d3a-41dbfa9e20f2-istio-data" pod "b39e7705-286b-4d9a-8d3a-41dbfa9e20f2" (UID: "b39e7705-286b-4d9a-8d3a-41dbfa9e20f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:40:15.199082 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.198288    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/033d7e0d-2790-46af-829c-78374de43cd1-workload-certs" pod "033d7e0d-2790-46af-829c-78374de43cd1" (UID: "033d7e0d-2790-46af-829c-78374de43cd1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:40:15.199082 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.198399    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/033d7e0d-2790-46af-829c-78374de43cd1-istiod-ca-cert" pod "033d7e0d-2790-46af-829c-78374de43cd1" (UID: "033d7e0d-2790-46af-829c-78374de43cd1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:40:15.199082 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.198563    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b39e7705-286b-4d9a-8d3a-41dbfa9e20f2-istiod-ca-cert" pod "b39e7705-286b-4d9a-8d3a-41dbfa9e20f2" (UID: "b39e7705-286b-4d9a-8d3a-41dbfa9e20f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:40:15.199082 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.198656    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b39e7705-286b-4d9a-8d3a-41dbfa9e20f2-workload-certs" pod "b39e7705-286b-4d9a-8d3a-41dbfa9e20f2" (UID: "b39e7705-286b-4d9a-8d3a-41dbfa9e20f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:40:15.199378 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.198769    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/033d7e0d-2790-46af-829c-78374de43cd1-istio-data" pod "033d7e0d-2790-46af-829c-78374de43cd1" (UID: "033d7e0d-2790-46af-829c-78374de43cd1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:40:15.199378 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.198847    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/033d7e0d-2790-46af-829c-78374de43cd1-credential-socket" pod "033d7e0d-2790-46af-829c-78374de43cd1" (UID: "033d7e0d-2790-46af-829c-78374de43cd1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:40:15.199378 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.198871    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/033d7e0d-2790-46af-829c-78374de43cd1-workload-socket" pod "033d7e0d-2790-46af-829c-78374de43cd1" (UID: "033d7e0d-2790-46af-829c-78374de43cd1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:40:15.199378 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.198957    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b39e7705-286b-4d9a-8d3a-41dbfa9e20f2-credential-socket" pod "b39e7705-286b-4d9a-8d3a-41dbfa9e20f2" (UID: "b39e7705-286b-4d9a-8d3a-41dbfa9e20f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:40:15.199378 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.198982    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b39e7705-286b-4d9a-8d3a-41dbfa9e20f2-workload-socket" pod "b39e7705-286b-4d9a-8d3a-41dbfa9e20f2" (UID: "b39e7705-286b-4d9a-8d3a-41dbfa9e20f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:40:15.199540 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.199394    5898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/033d7e0d-2790-46af-829c-78374de43cd1-istiod-ca-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:15.199540 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.199414    5898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/033d7e0d-2790-46af-829c-78374de43cd1-istio-data\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:15.199540 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.199428    5898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b39e7705-286b-4d9a-8d3a-41dbfa9e20f2-istiod-ca-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:15.199540 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.199443    5898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/033d7e0d-2790-46af-829c-78374de43cd1-workload-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:15.199540 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.199457    5898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39e7705-286b-4d9a-8d3a-41dbfa9e20f2-workload-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:15.199540 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.199470    5898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39e7705-286b-4d9a-8d3a-41dbfa9e20f2-credential-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:15.199540 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.199484    5898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b39e7705-286b-4d9a-8d3a-41dbfa9e20f2-istio-data\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:15.199540 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.199498    5898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/033d7e0d-2790-46af-829c-78374de43cd1-credential-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:15.199540 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.199510    5898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b39e7705-286b-4d9a-8d3a-41dbfa9e20f2-workload-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:15.199540 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.199523    5898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/033d7e0d-2790-46af-829c-78374de43cd1-workload-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:15.200955 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.200896    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b39e7705-286b-4d9a-8d3a-41dbfa9e20f2-istio-envoy" pod "b39e7705-286b-4d9a-8d3a-41dbfa9e20f2" (UID: "b39e7705-286b-4d9a-8d3a-41dbfa9e20f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:40:15.204185 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.204160    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/033d7e0d-2790-46af-829c-78374de43cd1-istio-token" pod "033d7e0d-2790-46af-829c-78374de43cd1" (UID: "033d7e0d-2790-46af-829c-78374de43cd1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:40:15.204660 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.204625    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/033d7e0d-2790-46af-829c-78374de43cd1-kube-api-access-hs2rh" pod "033d7e0d-2790-46af-829c-78374de43cd1" (UID: "033d7e0d-2790-46af-829c-78374de43cd1"). InnerVolumeSpecName "kube-api-access-hs2rh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:40:15.204660 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.204654    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b39e7705-286b-4d9a-8d3a-41dbfa9e20f2-istio-token" pod "b39e7705-286b-4d9a-8d3a-41dbfa9e20f2" (UID: "b39e7705-286b-4d9a-8d3a-41dbfa9e20f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:40:15.204776 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.204664    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/033d7e0d-2790-46af-829c-78374de43cd1-istio-podinfo" pod "033d7e0d-2790-46af-829c-78374de43cd1" (UID: "033d7e0d-2790-46af-829c-78374de43cd1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:40:15.207697 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.207663    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b39e7705-286b-4d9a-8d3a-41dbfa9e20f2-istio-podinfo" pod "b39e7705-286b-4d9a-8d3a-41dbfa9e20f2" (UID: "b39e7705-286b-4d9a-8d3a-41dbfa9e20f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:40:15.207807 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.207759    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b39e7705-286b-4d9a-8d3a-41dbfa9e20f2-kube-api-access-k4pzq" pod "b39e7705-286b-4d9a-8d3a-41dbfa9e20f2" (UID: "b39e7705-286b-4d9a-8d3a-41dbfa9e20f2"). InnerVolumeSpecName "kube-api-access-k4pzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:40:15.207871 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.207815    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/033d7e0d-2790-46af-829c-78374de43cd1-istio-envoy" pod "033d7e0d-2790-46af-829c-78374de43cd1" (UID: "033d7e0d-2790-46af-829c-78374de43cd1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:40:15.300863 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.300820    5898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b39e7705-286b-4d9a-8d3a-41dbfa9e20f2-istio-token\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:15.301080 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.300868    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs2rh\" (UniqueName: \"kubernetes.io/projected/033d7e0d-2790-46af-829c-78374de43cd1-kube-api-access-hs2rh\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:15.301080 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.300886    5898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/033d7e0d-2790-46af-829c-78374de43cd1-istio-envoy\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:15.301080 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.300932    5898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b39e7705-286b-4d9a-8d3a-41dbfa9e20f2-istio-envoy\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:15.301080 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.300947    5898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/033d7e0d-2790-46af-829c-78374de43cd1-istio-podinfo\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:15.301080 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.300960    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4pzq\" (UniqueName: \"kubernetes.io/projected/b39e7705-286b-4d9a-8d3a-41dbfa9e20f2-kube-api-access-k4pzq\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:15.301080 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.300975    5898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b39e7705-286b-4d9a-8d3a-41dbfa9e20f2-istio-podinfo\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:15.301080 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.300990    5898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/033d7e0d-2790-46af-829c-78374de43cd1-istio-token\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:15.353826 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.353747    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5"]
Oct 30 05:40:15.497132 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.497101    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:40:15.643678 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.643589    5898 generic.go:356] "Generic (PLEG): container finished" podID="b39e7705-286b-4d9a-8d3a-41dbfa9e20f2" containerID="62cce9c2945e944d0b4851cc15b0904e970c4a8ee45585b94354f5f6b026113c" exitCode=0
Oct 30 05:40:15.643678 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.643638    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7"
Oct 30 05:40:15.644252 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.643681    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7" event={"ID":"b39e7705-286b-4d9a-8d3a-41dbfa9e20f2","Type":"ContainerDied","Data":"62cce9c2945e944d0b4851cc15b0904e970c4a8ee45585b94354f5f6b026113c"}
Oct 30 05:40:15.644252 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.643753    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7" event={"ID":"b39e7705-286b-4d9a-8d3a-41dbfa9e20f2","Type":"ContainerDied","Data":"c00fa7e5a1104f94bf2259f698893beaef8541da477c84aba66d1f72ccf6b16f"}
Oct 30 05:40:15.644252 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.643790    5898 scope.go:122] "RemoveContainer" containerID="62cce9c2945e944d0b4851cc15b0904e970c4a8ee45585b94354f5f6b026113c"
Oct 30 05:40:15.645640 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.645614    5898 generic.go:356] "Generic (PLEG): container finished" podID="033d7e0d-2790-46af-829c-78374de43cd1" containerID="2143370d428514c87be5c439b81b6fcd172b9051c43146d903a849096688c2f7" exitCode=0
Oct 30 05:40:15.645787 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.645651    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54" event={"ID":"033d7e0d-2790-46af-829c-78374de43cd1","Type":"ContainerDied","Data":"2143370d428514c87be5c439b81b6fcd172b9051c43146d903a849096688c2f7"}
Oct 30 05:40:15.645787 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.645694    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54" event={"ID":"033d7e0d-2790-46af-829c-78374de43cd1","Type":"ContainerDied","Data":"a816dd887f1492ff6e042edce2b47163af3595d070a7ddfb182ac8b75d2f6010"}
Oct 30 05:40:15.645787 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.645764    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54"
Oct 30 05:40:15.661872 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.661852    5898 scope.go:122] "RemoveContainer" containerID="62cce9c2945e944d0b4851cc15b0904e970c4a8ee45585b94354f5f6b026113c"
Oct 30 05:40:15.662636 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:40:15.662600    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62cce9c2945e944d0b4851cc15b0904e970c4a8ee45585b94354f5f6b026113c\": container with ID starting with 62cce9c2945e944d0b4851cc15b0904e970c4a8ee45585b94354f5f6b026113c not found: ID does not exist" containerID="62cce9c2945e944d0b4851cc15b0904e970c4a8ee45585b94354f5f6b026113c"
Oct 30 05:40:15.662770 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.662642    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62cce9c2945e944d0b4851cc15b0904e970c4a8ee45585b94354f5f6b026113c"} err="failed to get container status \"62cce9c2945e944d0b4851cc15b0904e970c4a8ee45585b94354f5f6b026113c\": rpc error: code = NotFound desc = could not find container \"62cce9c2945e944d0b4851cc15b0904e970c4a8ee45585b94354f5f6b026113c\": container with ID starting with 62cce9c2945e944d0b4851cc15b0904e970c4a8ee45585b94354f5f6b026113c not found: ID does not exist"
Oct 30 05:40:15.662770 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.662663    5898 scope.go:122] "RemoveContainer" containerID="2143370d428514c87be5c439b81b6fcd172b9051c43146d903a849096688c2f7"
Oct 30 05:40:15.672477 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.672455    5898 scope.go:122] "RemoveContainer" containerID="2143370d428514c87be5c439b81b6fcd172b9051c43146d903a849096688c2f7"
Oct 30 05:40:15.672745 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:40:15.672726    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2143370d428514c87be5c439b81b6fcd172b9051c43146d903a849096688c2f7\": container with ID starting with 2143370d428514c87be5c439b81b6fcd172b9051c43146d903a849096688c2f7 not found: ID does not exist" containerID="2143370d428514c87be5c439b81b6fcd172b9051c43146d903a849096688c2f7"
Oct 30 05:40:15.672815 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.672753    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2143370d428514c87be5c439b81b6fcd172b9051c43146d903a849096688c2f7"} err="failed to get container status \"2143370d428514c87be5c439b81b6fcd172b9051c43146d903a849096688c2f7\": rpc error: code = NotFound desc = could not find container \"2143370d428514c87be5c439b81b6fcd172b9051c43146d903a849096688c2f7\": container with ID starting with 2143370d428514c87be5c439b81b6fcd172b9051c43146d903a849096688c2f7 not found: ID does not exist"
Oct 30 05:40:15.722741 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.722698    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54"]
Oct 30 05:40:15.739254 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.739215    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-mcm54"]
Oct 30 05:40:15.803085 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.803041    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7"]
Oct 30 05:40:15.814602 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:15.814570    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9dc17a-openshift-default-74c47fbc45-kv9x7"]
Oct 30 05:40:16.224218 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:16.224185    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:40:16.889314 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:16.889276    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="033d7e0d-2790-46af-829c-78374de43cd1" path="/var/lib/kubelet/pods/033d7e0d-2790-46af-829c-78374de43cd1/volumes"
Oct 30 05:40:16.889734 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:16.889720    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b39e7705-286b-4d9a-8d3a-41dbfa9e20f2" path="/var/lib/kubelet/pods/b39e7705-286b-4d9a-8d3a-41dbfa9e20f2/volumes"
Oct 30 05:40:17.999421 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:17.999386    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:40:20.008457 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:20.008425    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:40:21.176410 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:21.176366    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:40:21.319520 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:21.319490    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:40:21.619033 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:21.618999    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1375b879-tnj9g"]
Oct 30 05:40:21.619368 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:21.619356    5898 cpu_manager.go:414] "containerMap: removing container" podUID="274129ce-5c11-4060-ab39-856dd097ffb1" containerName="revoke-keys"
Oct 30 05:40:21.619412 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:21.619370    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="274129ce-5c11-4060-ab39-856dd097ffb1" containerName="revoke-keys"
Oct 30 05:40:21.619412 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:21.619392    5898 cpu_manager.go:414] "containerMap: removing container" podUID="24c6b2d4-3377-4079-8273-3f48064c3a98" containerName="main"
Oct 30 05:40:21.619412 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:21.619397    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24c6b2d4-3377-4079-8273-3f48064c3a98" containerName="main"
Oct 30 05:40:21.619412 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:21.619406    5898 cpu_manager.go:414] "containerMap: removing container" podUID="a9707f50-d5e1-409f-b8f3-c58629677aaf" containerName="main"
Oct 30 05:40:21.619412 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:21.619411    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9707f50-d5e1-409f-b8f3-c58629677aaf" containerName="main"
Oct 30 05:40:21.619646 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:21.619424    5898 cpu_manager.go:414] "containerMap: removing container" podUID="d30a7a86-f1da-451c-81dd-ebdbaeb4f8af" containerName="main"
Oct 30 05:40:21.619646 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:21.619431    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d30a7a86-f1da-451c-81dd-ebdbaeb4f8af" containerName="main"
Oct 30 05:40:21.619646 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:21.619449    5898 cpu_manager.go:414] "containerMap: removing container" podUID="b39e7705-286b-4d9a-8d3a-41dbfa9e20f2" containerName="istio-proxy"
Oct 30 05:40:21.619646 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:21.619454    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b39e7705-286b-4d9a-8d3a-41dbfa9e20f2" containerName="istio-proxy"
Oct 30 05:40:21.619646 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:21.619461    5898 cpu_manager.go:414] "containerMap: removing container" podUID="033d7e0d-2790-46af-829c-78374de43cd1" containerName="istio-proxy"
Oct 30 05:40:21.619646 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:21.619466    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="033d7e0d-2790-46af-829c-78374de43cd1" containerName="istio-proxy"
Oct 30 05:40:21.619646 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:21.619515    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="24c6b2d4-3377-4079-8273-3f48064c3a98" containerName="main"
Oct 30 05:40:21.619646 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:21.619523    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9707f50-d5e1-409f-b8f3-c58629677aaf" containerName="main"
Oct 30 05:40:21.619646 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:21.619531    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="d30a7a86-f1da-451c-81dd-ebdbaeb4f8af" containerName="main"
Oct 30 05:40:21.619646 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:21.619539    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="b39e7705-286b-4d9a-8d3a-41dbfa9e20f2" containerName="istio-proxy"
Oct 30 05:40:21.619646 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:21.619545    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="033d7e0d-2790-46af-829c-78374de43cd1" containerName="istio-proxy"
Oct 30 05:40:21.619646 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:21.619551    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="274129ce-5c11-4060-ab39-856dd097ffb1" containerName="revoke-keys"
Oct 30 05:40:21.624299 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:21.624281    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1375b879-tnj9g"
Oct 30 05:40:21.656980 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:21.656860    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1375b879-tnj9g"]
Oct 30 05:40:21.708118 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:21.708085    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\""
Oct 30 05:40:21.708317 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:21.708235    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:40:21.718573 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:21.718547    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 05:40:21.718896 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:21.718879    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 05:40:21.768393 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:21.768352    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19904b19-a70f-4ec6-b99e-94b153f58f67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1375b879-tnj9g\" (UID: \"19904b19-a70f-4ec6-b99e-94b153f58f67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1375b879-tnj9g"
Oct 30 05:40:21.768566 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:21.768409    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z65f4\" (UniqueName: \"kubernetes.io/projected/19904b19-a70f-4ec6-b99e-94b153f58f67-kube-api-access-z65f4\") pod \"maas-api-revoke-sub-1375b879-tnj9g\" (UID: \"19904b19-a70f-4ec6-b99e-94b153f58f67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1375b879-tnj9g"
Oct 30 05:40:21.768566 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:21.768490    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19904b19-a70f-4ec6-b99e-94b153f58f67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1375b879-tnj9g\" (UID: \"19904b19-a70f-4ec6-b99e-94b153f58f67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1375b879-tnj9g"
Oct 30 05:40:21.869688 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:21.869607    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19904b19-a70f-4ec6-b99e-94b153f58f67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1375b879-tnj9g\" (UID: \"19904b19-a70f-4ec6-b99e-94b153f58f67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1375b879-tnj9g"
Oct 30 05:40:21.869688 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:21.869651    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z65f4\" (UniqueName: \"kubernetes.io/projected/19904b19-a70f-4ec6-b99e-94b153f58f67-kube-api-access-z65f4\") pod \"maas-api-revoke-sub-1375b879-tnj9g\" (UID: \"19904b19-a70f-4ec6-b99e-94b153f58f67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1375b879-tnj9g"
Oct 30 05:40:21.869688 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:21.869672    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19904b19-a70f-4ec6-b99e-94b153f58f67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1375b879-tnj9g\" (UID: \"19904b19-a70f-4ec6-b99e-94b153f58f67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1375b879-tnj9g"
Oct 30 05:40:21.870319 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:21.870299    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19904b19-a70f-4ec6-b99e-94b153f58f67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1375b879-tnj9g\" (UID: \"19904b19-a70f-4ec6-b99e-94b153f58f67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1375b879-tnj9g"
Oct 30 05:40:21.872146 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:21.872113    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19904b19-a70f-4ec6-b99e-94b153f58f67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1375b879-tnj9g\" (UID: \"19904b19-a70f-4ec6-b99e-94b153f58f67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1375b879-tnj9g"
Oct 30 05:40:21.889143 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:21.889121    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z65f4\" (UniqueName: \"kubernetes.io/projected/19904b19-a70f-4ec6-b99e-94b153f58f67-kube-api-access-z65f4\") pod \"maas-api-revoke-sub-1375b879-tnj9g\" (UID: \"19904b19-a70f-4ec6-b99e-94b153f58f67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1375b879-tnj9g"
Oct 30 05:40:21.936725 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:21.936698    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1375b879-tnj9g"
Oct 30 05:40:22.252009 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:22.251929    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1375b879-tnj9g"]
Oct 30 05:40:22.256637 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:40:22.256609    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19904b19_a70f_4ec6_b99e_94b153f58f67.slice/crio-97b6c3c1fa8651132e6e60a2778d5e18d5829fd671382f4dcb23c03416e852a6 WatchSource:0}: Error finding container 97b6c3c1fa8651132e6e60a2778d5e18d5829fd671382f4dcb23c03416e852a6: Status 404 returned error can't find the container with id 97b6c3c1fa8651132e6e60a2778d5e18d5829fd671382f4dcb23c03416e852a6
Oct 30 05:40:22.283191 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:22.283169    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:40:22.402955 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:22.402922    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:40:22.604817 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:22.598754    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw"]
Oct 30 05:40:22.604817 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:22.599004    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw" podUID="8cdfbc2d-3a92-4782-89cb-886f47416486" containerName="istio-proxy" containerID="cri-o://d9adc65b9f0cc67eb73c801f80309adff85411bda04fb280a41252f6461875db" gracePeriod=30
Oct 30 05:40:22.659697 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:22.659659    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:40:22.728158 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:22.728062    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1375b879-tnj9g" event={"ID":"19904b19-a70f-4ec6-b99e-94b153f58f67","Type":"ContainerStarted","Data":"a18666e716209aab884c1d15d6ffd1943afe544a337cce54d82656c739de6657"}
Oct 30 05:40:22.728533 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:22.728506    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1375b879-tnj9g" event={"ID":"19904b19-a70f-4ec6-b99e-94b153f58f67","Type":"ContainerStarted","Data":"97b6c3c1fa8651132e6e60a2778d5e18d5829fd671382f4dcb23c03416e852a6"}
Oct 30 05:40:22.774252 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:22.774189    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1375b879-tnj9g" podStartSLOduration=1.7741690719999998 podStartE2EDuration="1.774169072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:40:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:40:22.772159717 +0000 UTC m=+3351.995979645" watchObservedRunningTime="2026-10-30 05:40:22.774169072 +0000 UTC m=+3351.997988974"
Oct 30 05:40:23.734933 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:23.734878    5898 generic.go:356] "Generic (PLEG): container finished" podID="19904b19-a70f-4ec6-b99e-94b153f58f67" containerID="a18666e716209aab884c1d15d6ffd1943afe544a337cce54d82656c739de6657" exitCode=0
Oct 30 05:40:23.735337 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:23.734965    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1375b879-tnj9g" event={"ID":"19904b19-a70f-4ec6-b99e-94b153f58f67","Type":"ContainerDied","Data":"a18666e716209aab884c1d15d6ffd1943afe544a337cce54d82656c739de6657"}
Oct 30 05:40:24.809707 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:24.809668    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:40:24.877795 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:24.877774    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1375b879-tnj9g"
Oct 30 05:40:25.003075 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:25.002996    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19904b19-a70f-4ec6-b99e-94b153f58f67-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19904b19-a70f-4ec6-b99e-94b153f58f67-maas-api-openshift-ca\") pod \"19904b19-a70f-4ec6-b99e-94b153f58f67\" (UID: \"19904b19-a70f-4ec6-b99e-94b153f58f67\") "
Oct 30 05:40:25.003075 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:25.003047    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19904b19-a70f-4ec6-b99e-94b153f58f67-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19904b19-a70f-4ec6-b99e-94b153f58f67-maas-api-xks-ca\") pod \"19904b19-a70f-4ec6-b99e-94b153f58f67\" (UID: \"19904b19-a70f-4ec6-b99e-94b153f58f67\") "
Oct 30 05:40:25.003285 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:25.003110    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19904b19-a70f-4ec6-b99e-94b153f58f67-kube-api-access-z65f4\" (UniqueName: \"kubernetes.io/projected/19904b19-a70f-4ec6-b99e-94b153f58f67-kube-api-access-z65f4\") pod \"19904b19-a70f-4ec6-b99e-94b153f58f67\" (UID: \"19904b19-a70f-4ec6-b99e-94b153f58f67\") "
Oct 30 05:40:25.003422 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:25.003400    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19904b19-a70f-4ec6-b99e-94b153f58f67-maas-api-openshift-ca" pod "19904b19-a70f-4ec6-b99e-94b153f58f67" (UID: "19904b19-a70f-4ec6-b99e-94b153f58f67"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:40:25.003483 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:25.003470    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19904b19-a70f-4ec6-b99e-94b153f58f67-maas-api-openshift-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:25.005164 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:25.005131    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19904b19-a70f-4ec6-b99e-94b153f58f67-maas-api-xks-ca" pod "19904b19-a70f-4ec6-b99e-94b153f58f67" (UID: "19904b19-a70f-4ec6-b99e-94b153f58f67"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:40:25.005288 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:25.005207    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19904b19-a70f-4ec6-b99e-94b153f58f67-kube-api-access-z65f4" pod "19904b19-a70f-4ec6-b99e-94b153f58f67" (UID: "19904b19-a70f-4ec6-b99e-94b153f58f67"). InnerVolumeSpecName "kube-api-access-z65f4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:40:25.104666 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:25.104633    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19904b19-a70f-4ec6-b99e-94b153f58f67-maas-api-xks-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:25.104666 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:25.104666    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z65f4\" (UniqueName: \"kubernetes.io/projected/19904b19-a70f-4ec6-b99e-94b153f58f67-kube-api-access-z65f4\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:25.723872 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:25.723840    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:40:25.755174 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:25.750685    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1375b879-tnj9g" event={"ID":"19904b19-a70f-4ec6-b99e-94b153f58f67","Type":"ContainerDied","Data":"97b6c3c1fa8651132e6e60a2778d5e18d5829fd671382f4dcb23c03416e852a6"}
Oct 30 05:40:25.755174 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:25.750750    5898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97b6c3c1fa8651132e6e60a2778d5e18d5829fd671382f4dcb23c03416e852a6"
Oct 30 05:40:25.755174 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:25.750818    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1375b879-tnj9g"
Oct 30 05:40:25.823585 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:25.823538    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:40:26.096637 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:26.096606    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:40:26.817328 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:40:26.817288    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:40:27.763658 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:27.763625    5898 generic.go:356] "Generic (PLEG): container finished" podID="8cdfbc2d-3a92-4782-89cb-886f47416486" containerID="d9adc65b9f0cc67eb73c801f80309adff85411bda04fb280a41252f6461875db" exitCode=0
Oct 30 05:40:27.764171 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:27.763708    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw" event={"ID":"8cdfbc2d-3a92-4782-89cb-886f47416486","Type":"ContainerDied","Data":"d9adc65b9f0cc67eb73c801f80309adff85411bda04fb280a41252f6461875db"}
Oct 30 05:40:27.848563 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:27.848533    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw"
Oct 30 05:40:27.930310 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:27.930226    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cdfbc2d-3a92-4782-89cb-886f47416486-istio-token\" (UniqueName: \"kubernetes.io/projected/8cdfbc2d-3a92-4782-89cb-886f47416486-istio-token\") pod \"8cdfbc2d-3a92-4782-89cb-886f47416486\" (UID: \"8cdfbc2d-3a92-4782-89cb-886f47416486\") "
Oct 30 05:40:27.930310 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:27.930276    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cdfbc2d-3a92-4782-89cb-886f47416486-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cdfbc2d-3a92-4782-89cb-886f47416486-credential-socket\") pod \"8cdfbc2d-3a92-4782-89cb-886f47416486\" (UID: \"8cdfbc2d-3a92-4782-89cb-886f47416486\") "
Oct 30 05:40:27.930573 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:27.930308    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cdfbc2d-3a92-4782-89cb-886f47416486-kube-api-access-7mfc2\" (UniqueName: \"kubernetes.io/projected/8cdfbc2d-3a92-4782-89cb-886f47416486-kube-api-access-7mfc2\") pod \"8cdfbc2d-3a92-4782-89cb-886f47416486\" (UID: \"8cdfbc2d-3a92-4782-89cb-886f47416486\") "
Oct 30 05:40:27.930573 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:27.930382    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cdfbc2d-3a92-4782-89cb-886f47416486-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cdfbc2d-3a92-4782-89cb-886f47416486-istio-envoy\") pod \"8cdfbc2d-3a92-4782-89cb-886f47416486\" (UID: \"8cdfbc2d-3a92-4782-89cb-886f47416486\") "
Oct 30 05:40:27.930573 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:27.930407    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cdfbc2d-3a92-4782-89cb-886f47416486-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cdfbc2d-3a92-4782-89cb-886f47416486-istiod-ca-cert\") pod \"8cdfbc2d-3a92-4782-89cb-886f47416486\" (UID: \"8cdfbc2d-3a92-4782-89cb-886f47416486\") "
Oct 30 05:40:27.930573 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:27.930438    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8cdfbc2d-3a92-4782-89cb-886f47416486-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cdfbc2d-3a92-4782-89cb-886f47416486-istio-podinfo\") pod \"8cdfbc2d-3a92-4782-89cb-886f47416486\" (UID: \"8cdfbc2d-3a92-4782-89cb-886f47416486\") "
Oct 30 05:40:27.930573 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:27.930495    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cdfbc2d-3a92-4782-89cb-886f47416486-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cdfbc2d-3a92-4782-89cb-886f47416486-workload-certs\") pod \"8cdfbc2d-3a92-4782-89cb-886f47416486\" (UID: \"8cdfbc2d-3a92-4782-89cb-886f47416486\") "
Oct 30 05:40:27.930573 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:27.930527    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cdfbc2d-3a92-4782-89cb-886f47416486-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cdfbc2d-3a92-4782-89cb-886f47416486-workload-socket\") pod \"8cdfbc2d-3a92-4782-89cb-886f47416486\" (UID: \"8cdfbc2d-3a92-4782-89cb-886f47416486\") "
Oct 30 05:40:27.930948 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:27.930588    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cdfbc2d-3a92-4782-89cb-886f47416486-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cdfbc2d-3a92-4782-89cb-886f47416486-istio-data\") pod \"8cdfbc2d-3a92-4782-89cb-886f47416486\" (UID: \"8cdfbc2d-3a92-4782-89cb-886f47416486\") "
Oct 30 05:40:27.931251 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:27.931208    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cdfbc2d-3a92-4782-89cb-886f47416486-istio-data" pod "8cdfbc2d-3a92-4782-89cb-886f47416486" (UID: "8cdfbc2d-3a92-4782-89cb-886f47416486"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:40:27.931620 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:27.931599    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cdfbc2d-3a92-4782-89cb-886f47416486-istiod-ca-cert" pod "8cdfbc2d-3a92-4782-89cb-886f47416486" (UID: "8cdfbc2d-3a92-4782-89cb-886f47416486"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:40:27.932322 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:27.932292    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cdfbc2d-3a92-4782-89cb-886f47416486-workload-certs" pod "8cdfbc2d-3a92-4782-89cb-886f47416486" (UID: "8cdfbc2d-3a92-4782-89cb-886f47416486"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:40:27.932428 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:27.932314    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cdfbc2d-3a92-4782-89cb-886f47416486-workload-socket" pod "8cdfbc2d-3a92-4782-89cb-886f47416486" (UID: "8cdfbc2d-3a92-4782-89cb-886f47416486"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:40:27.932428 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:27.932387    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cdfbc2d-3a92-4782-89cb-886f47416486-credential-socket" pod "8cdfbc2d-3a92-4782-89cb-886f47416486" (UID: "8cdfbc2d-3a92-4782-89cb-886f47416486"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:40:27.933864 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:27.933838    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cdfbc2d-3a92-4782-89cb-886f47416486-istio-token" pod "8cdfbc2d-3a92-4782-89cb-886f47416486" (UID: "8cdfbc2d-3a92-4782-89cb-886f47416486"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:40:27.934046 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:27.933971    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cdfbc2d-3a92-4782-89cb-886f47416486-kube-api-access-7mfc2" pod "8cdfbc2d-3a92-4782-89cb-886f47416486" (UID: "8cdfbc2d-3a92-4782-89cb-886f47416486"). InnerVolumeSpecName "kube-api-access-7mfc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:40:27.934131 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:27.934047    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cdfbc2d-3a92-4782-89cb-886f47416486-istio-envoy" pod "8cdfbc2d-3a92-4782-89cb-886f47416486" (UID: "8cdfbc2d-3a92-4782-89cb-886f47416486"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:40:27.934131 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:27.934072    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8cdfbc2d-3a92-4782-89cb-886f47416486-istio-podinfo" pod "8cdfbc2d-3a92-4782-89cb-886f47416486" (UID: "8cdfbc2d-3a92-4782-89cb-886f47416486"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:40:28.031665 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:28.031632    5898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cdfbc2d-3a92-4782-89cb-886f47416486-istiod-ca-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:28.031665 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:28.031660    5898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cdfbc2d-3a92-4782-89cb-886f47416486-istio-podinfo\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:28.031665 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:28.031668    5898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cdfbc2d-3a92-4782-89cb-886f47416486-workload-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:28.031885 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:28.031677    5898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cdfbc2d-3a92-4782-89cb-886f47416486-workload-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:28.031885 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:28.031685    5898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cdfbc2d-3a92-4782-89cb-886f47416486-istio-data\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:28.031885 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:28.031693    5898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cdfbc2d-3a92-4782-89cb-886f47416486-istio-token\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:28.031885 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:28.031701    5898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cdfbc2d-3a92-4782-89cb-886f47416486-credential-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:28.031885 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:28.031710    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mfc2\" (UniqueName: \"kubernetes.io/projected/8cdfbc2d-3a92-4782-89cb-886f47416486-kube-api-access-7mfc2\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:28.031885 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:28.031719    5898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cdfbc2d-3a92-4782-89cb-886f47416486-istio-envoy\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:28.770618 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:28.770589    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw"
Oct 30 05:40:28.771069 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:28.770632    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw" event={"ID":"8cdfbc2d-3a92-4782-89cb-886f47416486","Type":"ContainerDied","Data":"9cd400d310cf1ce3c36d2f7c15458dc176142622d03b45317d5243364c325ed0"}
Oct 30 05:40:28.771069 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:28.770699    5898 scope.go:122] "RemoveContainer" containerID="d9adc65b9f0cc67eb73c801f80309adff85411bda04fb280a41252f6461875db"
Oct 30 05:40:28.833242 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:28.833198    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw"]
Oct 30 05:40:28.840738 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:28.840710    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-0b21ba-openshift-default-7776bf6747-zjkvw"]
Oct 30 05:40:28.888591 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:28.888561    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cdfbc2d-3a92-4782-89cb-886f47416486" path="/var/lib/kubelet/pods/8cdfbc2d-3a92-4782-89cb-886f47416486/volumes"
Oct 30 05:40:30.374269 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:30.374228    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5" podUID="ebf8f4a2-8ac6-48ec-bdba-3f6a99653264" containerName="main" containerID="cri-o://56cd3db553627014ce617bcbdab0f5dc61d13d2be7b18028f5c557283fc8e56b" gracePeriod=45
Oct 30 05:40:30.494684 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:30.494649    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5" podUID="ebf8f4a2-8ac6-48ec-bdba-3f6a99653264" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": dial tcp 10.133.0.52:8000: connect: connection refused"
Oct 30 05:40:30.628980 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:30.628922    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5"
Oct 30 05:40:30.653022 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:30.652995    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebf8f4a2-8ac6-48ec-bdba-3f6a99653264-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebf8f4a2-8ac6-48ec-bdba-3f6a99653264-model-cache\") pod \"ebf8f4a2-8ac6-48ec-bdba-3f6a99653264\" (UID: \"ebf8f4a2-8ac6-48ec-bdba-3f6a99653264\") "
Oct 30 05:40:30.653165 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:30.653038    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebf8f4a2-8ac6-48ec-bdba-3f6a99653264-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebf8f4a2-8ac6-48ec-bdba-3f6a99653264-dshm\") pod \"ebf8f4a2-8ac6-48ec-bdba-3f6a99653264\" (UID: \"ebf8f4a2-8ac6-48ec-bdba-3f6a99653264\") "
Oct 30 05:40:30.653165 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:30.653071    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebf8f4a2-8ac6-48ec-bdba-3f6a99653264-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebf8f4a2-8ac6-48ec-bdba-3f6a99653264-tmp-dir\") pod \"ebf8f4a2-8ac6-48ec-bdba-3f6a99653264\" (UID: \"ebf8f4a2-8ac6-48ec-bdba-3f6a99653264\") "
Oct 30 05:40:30.653165 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:30.653124    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebf8f4a2-8ac6-48ec-bdba-3f6a99653264-tls-certs\" (UniqueName: \"kubernetes.io/secret/ebf8f4a2-8ac6-48ec-bdba-3f6a99653264-tls-certs\") pod \"ebf8f4a2-8ac6-48ec-bdba-3f6a99653264\" (UID: \"ebf8f4a2-8ac6-48ec-bdba-3f6a99653264\") "
Oct 30 05:40:30.653348 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:30.653210    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebf8f4a2-8ac6-48ec-bdba-3f6a99653264-kube-api-access-255jw\" (UniqueName: \"kubernetes.io/projected/ebf8f4a2-8ac6-48ec-bdba-3f6a99653264-kube-api-access-255jw\") pod \"ebf8f4a2-8ac6-48ec-bdba-3f6a99653264\" (UID: \"ebf8f4a2-8ac6-48ec-bdba-3f6a99653264\") "
Oct 30 05:40:30.653348 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:30.653263    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebf8f4a2-8ac6-48ec-bdba-3f6a99653264-home\" (UniqueName: \"kubernetes.io/empty-dir/ebf8f4a2-8ac6-48ec-bdba-3f6a99653264-home\") pod \"ebf8f4a2-8ac6-48ec-bdba-3f6a99653264\" (UID: \"ebf8f4a2-8ac6-48ec-bdba-3f6a99653264\") "
Oct 30 05:40:30.653348 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:30.653299    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebf8f4a2-8ac6-48ec-bdba-3f6a99653264-model-cache" pod "ebf8f4a2-8ac6-48ec-bdba-3f6a99653264" (UID: "ebf8f4a2-8ac6-48ec-bdba-3f6a99653264"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:40:30.653348 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:30.653311    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebf8f4a2-8ac6-48ec-bdba-3f6a99653264-tmp-dir" pod "ebf8f4a2-8ac6-48ec-bdba-3f6a99653264" (UID: "ebf8f4a2-8ac6-48ec-bdba-3f6a99653264"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:40:30.653553 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:30.653516    5898 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebf8f4a2-8ac6-48ec-bdba-3f6a99653264-model-cache\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:30.653553 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:30.653529    5898 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebf8f4a2-8ac6-48ec-bdba-3f6a99653264-tmp-dir\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:30.653553 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:30.653526    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebf8f4a2-8ac6-48ec-bdba-3f6a99653264-home" pod "ebf8f4a2-8ac6-48ec-bdba-3f6a99653264" (UID: "ebf8f4a2-8ac6-48ec-bdba-3f6a99653264"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:40:30.655476 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:30.655454    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebf8f4a2-8ac6-48ec-bdba-3f6a99653264-dshm" pod "ebf8f4a2-8ac6-48ec-bdba-3f6a99653264" (UID: "ebf8f4a2-8ac6-48ec-bdba-3f6a99653264"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:40:30.655476 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:30.655462    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebf8f4a2-8ac6-48ec-bdba-3f6a99653264-tls-certs" pod "ebf8f4a2-8ac6-48ec-bdba-3f6a99653264" (UID: "ebf8f4a2-8ac6-48ec-bdba-3f6a99653264"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:40:30.655662 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:30.655510    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebf8f4a2-8ac6-48ec-bdba-3f6a99653264-kube-api-access-255jw" pod "ebf8f4a2-8ac6-48ec-bdba-3f6a99653264" (UID: "ebf8f4a2-8ac6-48ec-bdba-3f6a99653264"). InnerVolumeSpecName "kube-api-access-255jw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:40:30.754549 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:30.754511    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-255jw\" (UniqueName: \"kubernetes.io/projected/ebf8f4a2-8ac6-48ec-bdba-3f6a99653264-kube-api-access-255jw\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:30.754549 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:30.754547    5898 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebf8f4a2-8ac6-48ec-bdba-3f6a99653264-home\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:30.754766 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:30.754562    5898 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebf8f4a2-8ac6-48ec-bdba-3f6a99653264-dshm\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:30.754766 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:30.754575    5898 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebf8f4a2-8ac6-48ec-bdba-3f6a99653264-tls-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:30.785248 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:30.785209    5898 generic.go:356] "Generic (PLEG): container finished" podID="ebf8f4a2-8ac6-48ec-bdba-3f6a99653264" containerID="56cd3db553627014ce617bcbdab0f5dc61d13d2be7b18028f5c557283fc8e56b" exitCode=0
Oct 30 05:40:30.785430 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:30.785257    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5"
Oct 30 05:40:30.785430 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:30.785254    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5" event={"ID":"ebf8f4a2-8ac6-48ec-bdba-3f6a99653264","Type":"ContainerDied","Data":"56cd3db553627014ce617bcbdab0f5dc61d13d2be7b18028f5c557283fc8e56b"}
Oct 30 05:40:30.785430 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:30.785311    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5" event={"ID":"ebf8f4a2-8ac6-48ec-bdba-3f6a99653264","Type":"ContainerDied","Data":"ca04a46dedd10ccfc75e70bc1b94b6e53db786d7763db20d400e9eec6cc89149"}
Oct 30 05:40:30.785430 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:30.785335    5898 scope.go:122] "RemoveContainer" containerID="56cd3db553627014ce617bcbdab0f5dc61d13d2be7b18028f5c557283fc8e56b"
Oct 30 05:40:30.797682 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:30.797664    5898 scope.go:122] "RemoveContainer" containerID="56cd3db553627014ce617bcbdab0f5dc61d13d2be7b18028f5c557283fc8e56b"
Oct 30 05:40:30.798019 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:40:30.797998    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56cd3db553627014ce617bcbdab0f5dc61d13d2be7b18028f5c557283fc8e56b\": container with ID starting with 56cd3db553627014ce617bcbdab0f5dc61d13d2be7b18028f5c557283fc8e56b not found: ID does not exist" containerID="56cd3db553627014ce617bcbdab0f5dc61d13d2be7b18028f5c557283fc8e56b"
Oct 30 05:40:30.798117 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:30.798026    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56cd3db553627014ce617bcbdab0f5dc61d13d2be7b18028f5c557283fc8e56b"} err="failed to get container status \"56cd3db553627014ce617bcbdab0f5dc61d13d2be7b18028f5c557283fc8e56b\": rpc error: code = NotFound desc = could not find container \"56cd3db553627014ce617bcbdab0f5dc61d13d2be7b18028f5c557283fc8e56b\": container with ID starting with 56cd3db553627014ce617bcbdab0f5dc61d13d2be7b18028f5c557283fc8e56b not found: ID does not exist"
Oct 30 05:40:30.833404 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:30.833366    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5"]
Oct 30 05:40:30.838940 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:30.838878    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-cff4ac/premium-w1-cff4ac-kserve-7b4db76fbd-75cn5"]
Oct 30 05:40:30.888917 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:30.888833    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebf8f4a2-8ac6-48ec-bdba-3f6a99653264" path="/var/lib/kubelet/pods/ebf8f4a2-8ac6-48ec-bdba-3f6a99653264/volumes"
Oct 30 05:40:31.933486 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:31.933454    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:40:34.191487 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:34.191454    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:40:36.831683 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:36.831637    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:40:37.313480 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:37.313378    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:40:37.416366 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:37.416303    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:40:37.964326 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:37.964293    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:40:40.389325 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:40.389288    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:40:40.722309 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:40.722230    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:40:41.849044 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:41.849010    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bx4nb"]
Oct 30 05:40:41.849461 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:41.849242    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bx4nb" podUID="b258ca46-73e4-403f-82f6-74e0cd489856" containerName="discovery" containerID="cri-o://be4d849712a6bdffff9bc9a4a6716a5c4a283ac634149d85ffab97f06b5d57ac" gracePeriod=30
Oct 30 05:40:42.090218 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:42.090193    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bx4nb"
Oct 30 05:40:42.158778 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:42.158698    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b258ca46-73e4-403f-82f6-74e0cd489856-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b258ca46-73e4-403f-82f6-74e0cd489856-local-certs\") pod \"b258ca46-73e4-403f-82f6-74e0cd489856\" (UID: \"b258ca46-73e4-403f-82f6-74e0cd489856\") "
Oct 30 05:40:42.158778 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:42.158771    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b258ca46-73e4-403f-82f6-74e0cd489856-istio-token\" (UniqueName: \"kubernetes.io/projected/b258ca46-73e4-403f-82f6-74e0cd489856-istio-token\") pod \"b258ca46-73e4-403f-82f6-74e0cd489856\" (UID: \"b258ca46-73e4-403f-82f6-74e0cd489856\") "
Oct 30 05:40:42.159056 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:42.158802    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b258ca46-73e4-403f-82f6-74e0cd489856-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b258ca46-73e4-403f-82f6-74e0cd489856-istio-kubeconfig\") pod \"b258ca46-73e4-403f-82f6-74e0cd489856\" (UID: \"b258ca46-73e4-403f-82f6-74e0cd489856\") "
Oct 30 05:40:42.159056 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:42.158841    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b258ca46-73e4-403f-82f6-74e0cd489856-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b258ca46-73e4-403f-82f6-74e0cd489856-istio-csr-ca-configmap\") pod \"b258ca46-73e4-403f-82f6-74e0cd489856\" (UID: \"b258ca46-73e4-403f-82f6-74e0cd489856\") "
Oct 30 05:40:42.159056 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:42.158891    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b258ca46-73e4-403f-82f6-74e0cd489856-cacerts\" (UniqueName: \"kubernetes.io/secret/b258ca46-73e4-403f-82f6-74e0cd489856-cacerts\") pod \"b258ca46-73e4-403f-82f6-74e0cd489856\" (UID: \"b258ca46-73e4-403f-82f6-74e0cd489856\") "
Oct 30 05:40:42.159056 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:42.158942    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b258ca46-73e4-403f-82f6-74e0cd489856-kube-api-access-xhj5v\" (UniqueName: \"kubernetes.io/projected/b258ca46-73e4-403f-82f6-74e0cd489856-kube-api-access-xhj5v\") pod \"b258ca46-73e4-403f-82f6-74e0cd489856\" (UID: \"b258ca46-73e4-403f-82f6-74e0cd489856\") "
Oct 30 05:40:42.159056 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:42.158992    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b258ca46-73e4-403f-82f6-74e0cd489856-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b258ca46-73e4-403f-82f6-74e0cd489856-istio-csr-dns-cert\") pod \"b258ca46-73e4-403f-82f6-74e0cd489856\" (UID: \"b258ca46-73e4-403f-82f6-74e0cd489856\") "
Oct 30 05:40:42.159335 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:42.159223    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b258ca46-73e4-403f-82f6-74e0cd489856-istio-csr-ca-configmap" pod "b258ca46-73e4-403f-82f6-74e0cd489856" (UID: "b258ca46-73e4-403f-82f6-74e0cd489856"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:40:42.159441 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:42.159419    5898 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b258ca46-73e4-403f-82f6-74e0cd489856-istio-csr-ca-configmap\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:42.161110 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:42.161073    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b258ca46-73e4-403f-82f6-74e0cd489856-local-certs" pod "b258ca46-73e4-403f-82f6-74e0cd489856" (UID: "b258ca46-73e4-403f-82f6-74e0cd489856"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:40:42.161552 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:42.161524    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b258ca46-73e4-403f-82f6-74e0cd489856-istio-kubeconfig" pod "b258ca46-73e4-403f-82f6-74e0cd489856" (UID: "b258ca46-73e4-403f-82f6-74e0cd489856"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:40:42.161696 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:42.161670    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b258ca46-73e4-403f-82f6-74e0cd489856-cacerts" pod "b258ca46-73e4-403f-82f6-74e0cd489856" (UID: "b258ca46-73e4-403f-82f6-74e0cd489856"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:40:42.161830 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:42.161757    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b258ca46-73e4-403f-82f6-74e0cd489856-kube-api-access-xhj5v" pod "b258ca46-73e4-403f-82f6-74e0cd489856" (UID: "b258ca46-73e4-403f-82f6-74e0cd489856"). InnerVolumeSpecName "kube-api-access-xhj5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:40:42.161830 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:42.161764    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b258ca46-73e4-403f-82f6-74e0cd489856-istio-csr-dns-cert" pod "b258ca46-73e4-403f-82f6-74e0cd489856" (UID: "b258ca46-73e4-403f-82f6-74e0cd489856"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:40:42.161830 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:42.161826    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b258ca46-73e4-403f-82f6-74e0cd489856-istio-token" pod "b258ca46-73e4-403f-82f6-74e0cd489856" (UID: "b258ca46-73e4-403f-82f6-74e0cd489856"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:40:42.260470 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:42.260440    5898 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b258ca46-73e4-403f-82f6-74e0cd489856-local-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:42.260470 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:42.260466    5898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b258ca46-73e4-403f-82f6-74e0cd489856-istio-token\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:42.260470 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:42.260475    5898 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b258ca46-73e4-403f-82f6-74e0cd489856-istio-kubeconfig\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:42.260702 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:42.260484    5898 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b258ca46-73e4-403f-82f6-74e0cd489856-cacerts\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:42.260702 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:42.260493    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhj5v\" (UniqueName: \"kubernetes.io/projected/b258ca46-73e4-403f-82f6-74e0cd489856-kube-api-access-xhj5v\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:42.260702 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:42.260502    5898 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b258ca46-73e4-403f-82f6-74e0cd489856-istio-csr-dns-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:42.850602 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:42.850563    5898 generic.go:356] "Generic (PLEG): container finished" podID="b258ca46-73e4-403f-82f6-74e0cd489856" containerID="be4d849712a6bdffff9bc9a4a6716a5c4a283ac634149d85ffab97f06b5d57ac" exitCode=0
Oct 30 05:40:42.851134 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:42.850632    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bx4nb"
Oct 30 05:40:42.851134 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:42.850636    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bx4nb" event={"ID":"b258ca46-73e4-403f-82f6-74e0cd489856","Type":"ContainerDied","Data":"be4d849712a6bdffff9bc9a4a6716a5c4a283ac634149d85ffab97f06b5d57ac"}
Oct 30 05:40:42.851134 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:42.850770    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bx4nb" event={"ID":"b258ca46-73e4-403f-82f6-74e0cd489856","Type":"ContainerDied","Data":"7e5c6e22a6b0b013524b83e1106c14d9b2420b0d0c829f6a12927a1cff8a77f2"}
Oct 30 05:40:42.851134 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:42.850789    5898 scope.go:122] "RemoveContainer" containerID="be4d849712a6bdffff9bc9a4a6716a5c4a283ac634149d85ffab97f06b5d57ac"
Oct 30 05:40:42.860294 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:42.860276    5898 scope.go:122] "RemoveContainer" containerID="be4d849712a6bdffff9bc9a4a6716a5c4a283ac634149d85ffab97f06b5d57ac"
Oct 30 05:40:42.860577 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:40:42.860558    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be4d849712a6bdffff9bc9a4a6716a5c4a283ac634149d85ffab97f06b5d57ac\": container with ID starting with be4d849712a6bdffff9bc9a4a6716a5c4a283ac634149d85ffab97f06b5d57ac not found: ID does not exist" containerID="be4d849712a6bdffff9bc9a4a6716a5c4a283ac634149d85ffab97f06b5d57ac"
Oct 30 05:40:42.860631 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:42.860587    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be4d849712a6bdffff9bc9a4a6716a5c4a283ac634149d85ffab97f06b5d57ac"} err="failed to get container status \"be4d849712a6bdffff9bc9a4a6716a5c4a283ac634149d85ffab97f06b5d57ac\": rpc error: code = NotFound desc = could not find container \"be4d849712a6bdffff9bc9a4a6716a5c4a283ac634149d85ffab97f06b5d57ac\": container with ID starting with be4d849712a6bdffff9bc9a4a6716a5c4a283ac634149d85ffab97f06b5d57ac not found: ID does not exist"
Oct 30 05:40:42.903362 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:42.903338    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bx4nb"]
Oct 30 05:40:42.914685 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:42.914656    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bx4nb"]
Oct 30 05:40:43.965438 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:43.965404    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:40:44.079896 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:44.079864    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:40:44.249498 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:44.249426    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:40:44.334716 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:44.334665    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp"]
Oct 30 05:40:44.335434 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:44.335400    5898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp" podUID="e97ea58e-dd00-4d95-b178-806306ff635c" containerName="istio-proxy" containerID="cri-o://c1d05aa65baccbee37a898d2ed41aa95a28d777b6b1c02dbf43fb125c02ccadb" gracePeriod=30
Oct 30 05:40:44.362351 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:44.362320    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:40:44.888712 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:44.888681    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b258ca46-73e4-403f-82f6-74e0cd489856" path="/var/lib/kubelet/pods/b258ca46-73e4-403f-82f6-74e0cd489856/volumes"
Oct 30 05:40:45.504621 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:45.504584    5898 patch_prober.go:29] interesting pod/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 05:40:45.505201 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:45.504651    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp" podUID="e97ea58e-dd00-4d95-b178-806306ff635c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 05:40:49.589478 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:49.589443    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp"
Oct 30 05:40:49.730203 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:49.730093    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e97ea58e-dd00-4d95-b178-806306ff635c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e97ea58e-dd00-4d95-b178-806306ff635c-credential-socket\") pod \"e97ea58e-dd00-4d95-b178-806306ff635c\" (UID: \"e97ea58e-dd00-4d95-b178-806306ff635c\") "
Oct 30 05:40:49.730203 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:49.730148    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e97ea58e-dd00-4d95-b178-806306ff635c-kube-api-access-w5vkx\" (UniqueName: \"kubernetes.io/projected/e97ea58e-dd00-4d95-b178-806306ff635c-kube-api-access-w5vkx\") pod \"e97ea58e-dd00-4d95-b178-806306ff635c\" (UID: \"e97ea58e-dd00-4d95-b178-806306ff635c\") "
Oct 30 05:40:49.730203 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:49.730177    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e97ea58e-dd00-4d95-b178-806306ff635c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e97ea58e-dd00-4d95-b178-806306ff635c-istiod-ca-cert\") pod \"e97ea58e-dd00-4d95-b178-806306ff635c\" (UID: \"e97ea58e-dd00-4d95-b178-806306ff635c\") "
Oct 30 05:40:49.730203 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:49.730201    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e97ea58e-dd00-4d95-b178-806306ff635c-istio-token\" (UniqueName: \"kubernetes.io/projected/e97ea58e-dd00-4d95-b178-806306ff635c-istio-token\") pod \"e97ea58e-dd00-4d95-b178-806306ff635c\" (UID: \"e97ea58e-dd00-4d95-b178-806306ff635c\") "
Oct 30 05:40:49.730599 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:49.730286    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e97ea58e-dd00-4d95-b178-806306ff635c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e97ea58e-dd00-4d95-b178-806306ff635c-workload-certs\") pod \"e97ea58e-dd00-4d95-b178-806306ff635c\" (UID: \"e97ea58e-dd00-4d95-b178-806306ff635c\") "
Oct 30 05:40:49.730599 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:49.730332    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e97ea58e-dd00-4d95-b178-806306ff635c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e97ea58e-dd00-4d95-b178-806306ff635c-workload-socket\") pod \"e97ea58e-dd00-4d95-b178-806306ff635c\" (UID: \"e97ea58e-dd00-4d95-b178-806306ff635c\") "
Oct 30 05:40:49.730599 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:49.730361    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e97ea58e-dd00-4d95-b178-806306ff635c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e97ea58e-dd00-4d95-b178-806306ff635c-istio-envoy\") pod \"e97ea58e-dd00-4d95-b178-806306ff635c\" (UID: \"e97ea58e-dd00-4d95-b178-806306ff635c\") "
Oct 30 05:40:49.730599 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:49.730437    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e97ea58e-dd00-4d95-b178-806306ff635c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e97ea58e-dd00-4d95-b178-806306ff635c-istio-data\") pod \"e97ea58e-dd00-4d95-b178-806306ff635c\" (UID: \"e97ea58e-dd00-4d95-b178-806306ff635c\") "
Oct 30 05:40:49.730599 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:49.730449    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e97ea58e-dd00-4d95-b178-806306ff635c-credential-socket" pod "e97ea58e-dd00-4d95-b178-806306ff635c" (UID: "e97ea58e-dd00-4d95-b178-806306ff635c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:40:49.730599 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:49.730493    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e97ea58e-dd00-4d95-b178-806306ff635c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e97ea58e-dd00-4d95-b178-806306ff635c-istio-podinfo\") pod \"e97ea58e-dd00-4d95-b178-806306ff635c\" (UID: \"e97ea58e-dd00-4d95-b178-806306ff635c\") "
Oct 30 05:40:49.731068 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:49.730722    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e97ea58e-dd00-4d95-b178-806306ff635c-workload-certs" pod "e97ea58e-dd00-4d95-b178-806306ff635c" (UID: "e97ea58e-dd00-4d95-b178-806306ff635c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:40:49.731068 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:49.730739    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e97ea58e-dd00-4d95-b178-806306ff635c-workload-socket" pod "e97ea58e-dd00-4d95-b178-806306ff635c" (UID: "e97ea58e-dd00-4d95-b178-806306ff635c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:40:49.731068 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:49.730768    5898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e97ea58e-dd00-4d95-b178-806306ff635c-credential-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:49.731068 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:49.730762    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e97ea58e-dd00-4d95-b178-806306ff635c-istiod-ca-cert" pod "e97ea58e-dd00-4d95-b178-806306ff635c" (UID: "e97ea58e-dd00-4d95-b178-806306ff635c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:40:49.731068 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:49.731021    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e97ea58e-dd00-4d95-b178-806306ff635c-istio-data" pod "e97ea58e-dd00-4d95-b178-806306ff635c" (UID: "e97ea58e-dd00-4d95-b178-806306ff635c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:40:49.732799 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:49.732774    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e97ea58e-dd00-4d95-b178-806306ff635c-istio-envoy" pod "e97ea58e-dd00-4d95-b178-806306ff635c" (UID: "e97ea58e-dd00-4d95-b178-806306ff635c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:40:49.732950 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:49.732809    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e97ea58e-dd00-4d95-b178-806306ff635c-istio-podinfo" pod "e97ea58e-dd00-4d95-b178-806306ff635c" (UID: "e97ea58e-dd00-4d95-b178-806306ff635c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:40:49.733033 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:49.732973    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e97ea58e-dd00-4d95-b178-806306ff635c-kube-api-access-w5vkx" pod "e97ea58e-dd00-4d95-b178-806306ff635c" (UID: "e97ea58e-dd00-4d95-b178-806306ff635c"). InnerVolumeSpecName "kube-api-access-w5vkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:40:49.733100 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:49.733086    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e97ea58e-dd00-4d95-b178-806306ff635c-istio-token" pod "e97ea58e-dd00-4d95-b178-806306ff635c" (UID: "e97ea58e-dd00-4d95-b178-806306ff635c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:40:49.831884 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:49.831842    5898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e97ea58e-dd00-4d95-b178-806306ff635c-istio-data\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:49.831884 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:49.831872    5898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e97ea58e-dd00-4d95-b178-806306ff635c-istio-podinfo\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:49.831884 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:49.831883    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5vkx\" (UniqueName: \"kubernetes.io/projected/e97ea58e-dd00-4d95-b178-806306ff635c-kube-api-access-w5vkx\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:49.831884 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:49.831896    5898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e97ea58e-dd00-4d95-b178-806306ff635c-istiod-ca-cert\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:49.832286 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:49.831917    5898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e97ea58e-dd00-4d95-b178-806306ff635c-istio-token\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:49.832286 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:49.831926    5898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e97ea58e-dd00-4d95-b178-806306ff635c-workload-certs\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:49.832286 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:49.831933    5898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e97ea58e-dd00-4d95-b178-806306ff635c-workload-socket\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:49.832286 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:49.831941    5898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e97ea58e-dd00-4d95-b178-806306ff635c-istio-envoy\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:40:49.893444 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:49.893406    5898 generic.go:356] "Generic (PLEG): container finished" podID="e97ea58e-dd00-4d95-b178-806306ff635c" containerID="c1d05aa65baccbee37a898d2ed41aa95a28d777b6b1c02dbf43fb125c02ccadb" exitCode=0
Oct 30 05:40:49.893612 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:49.893452    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp"
Oct 30 05:40:49.893612 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:49.893457    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp" event={"ID":"e97ea58e-dd00-4d95-b178-806306ff635c","Type":"ContainerDied","Data":"c1d05aa65baccbee37a898d2ed41aa95a28d777b6b1c02dbf43fb125c02ccadb"}
Oct 30 05:40:49.893612 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:49.893497    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp" event={"ID":"e97ea58e-dd00-4d95-b178-806306ff635c","Type":"ContainerDied","Data":"c2e1e3c14e92a27aa451be71126a81563060cdd7f053ba4a5622829ce78d9c14"}
Oct 30 05:40:49.893612 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:49.893518    5898 scope.go:122] "RemoveContainer" containerID="c1d05aa65baccbee37a898d2ed41aa95a28d777b6b1c02dbf43fb125c02ccadb"
Oct 30 05:40:49.904528 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:49.904483    5898 scope.go:122] "RemoveContainer" containerID="c1d05aa65baccbee37a898d2ed41aa95a28d777b6b1c02dbf43fb125c02ccadb"
Oct 30 05:40:49.904801 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:40:49.904783    5898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1d05aa65baccbee37a898d2ed41aa95a28d777b6b1c02dbf43fb125c02ccadb\": container with ID starting with c1d05aa65baccbee37a898d2ed41aa95a28d777b6b1c02dbf43fb125c02ccadb not found: ID does not exist" containerID="c1d05aa65baccbee37a898d2ed41aa95a28d777b6b1c02dbf43fb125c02ccadb"
Oct 30 05:40:49.904887 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:49.904808    5898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1d05aa65baccbee37a898d2ed41aa95a28d777b6b1c02dbf43fb125c02ccadb"} err="failed to get container status \"c1d05aa65baccbee37a898d2ed41aa95a28d777b6b1c02dbf43fb125c02ccadb\": rpc error: code = NotFound desc = could not find container \"c1d05aa65baccbee37a898d2ed41aa95a28d777b6b1c02dbf43fb125c02ccadb\": container with ID starting with c1d05aa65baccbee37a898d2ed41aa95a28d777b6b1c02dbf43fb125c02ccadb not found: ID does not exist"
Oct 30 05:40:49.943730 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:49.943697    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp"]
Oct 30 05:40:49.953705 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:49.953681    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-cff4ac-openshift-default-75448d8ddf-w6slp"]
Oct 30 05:40:50.890107 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:50.890069    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e97ea58e-dd00-4d95-b178-806306ff635c" path="/var/lib/kubelet/pods/e97ea58e-dd00-4d95-b178-806306ff635c/volumes"
Oct 30 05:40:52.221057 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:52.221030    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:40:52.464101 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:52.464069    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:40:52.732548 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:52.732514    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:40:52.860997 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:52.860963    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:40:53.854682 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:53.854624    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:40:56.819663 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:40:56.819611    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:40:59.362162 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:40:59.362131    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:41:04.705359 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:04.705322    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x"]
Oct 30 05:41:04.705866 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:04.705681    5898 cpu_manager.go:414] "containerMap: removing container" podUID="e97ea58e-dd00-4d95-b178-806306ff635c" containerName="istio-proxy"
Oct 30 05:41:04.705866 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:04.705692    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e97ea58e-dd00-4d95-b178-806306ff635c" containerName="istio-proxy"
Oct 30 05:41:04.705866 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:04.705709    5898 cpu_manager.go:414] "containerMap: removing container" podUID="b258ca46-73e4-403f-82f6-74e0cd489856" containerName="discovery"
Oct 30 05:41:04.705866 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:04.705714    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b258ca46-73e4-403f-82f6-74e0cd489856" containerName="discovery"
Oct 30 05:41:04.705866 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:04.705729    5898 cpu_manager.go:414] "containerMap: removing container" podUID="8cdfbc2d-3a92-4782-89cb-886f47416486" containerName="istio-proxy"
Oct 30 05:41:04.705866 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:04.705735    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cdfbc2d-3a92-4782-89cb-886f47416486" containerName="istio-proxy"
Oct 30 05:41:04.705866 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:04.705750    5898 cpu_manager.go:414] "containerMap: removing container" podUID="ebf8f4a2-8ac6-48ec-bdba-3f6a99653264" containerName="main"
Oct 30 05:41:04.705866 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:04.705754    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebf8f4a2-8ac6-48ec-bdba-3f6a99653264" containerName="main"
Oct 30 05:41:04.705866 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:04.705761    5898 cpu_manager.go:414] "containerMap: removing container" podUID="19904b19-a70f-4ec6-b99e-94b153f58f67" containerName="revoke-keys"
Oct 30 05:41:04.705866 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:04.705765    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19904b19-a70f-4ec6-b99e-94b153f58f67" containerName="revoke-keys"
Oct 30 05:41:04.705866 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:04.705819    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="e97ea58e-dd00-4d95-b178-806306ff635c" containerName="istio-proxy"
Oct 30 05:41:04.705866 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:04.705829    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="b258ca46-73e4-403f-82f6-74e0cd489856" containerName="discovery"
Oct 30 05:41:04.705866 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:04.705836    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cdfbc2d-3a92-4782-89cb-886f47416486" containerName="istio-proxy"
Oct 30 05:41:04.705866 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:04.705844    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebf8f4a2-8ac6-48ec-bdba-3f6a99653264" containerName="main"
Oct 30 05:41:04.705866 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:04.705848    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="19904b19-a70f-4ec6-b99e-94b153f58f67" containerName="revoke-keys"
Oct 30 05:41:04.709851 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:04.709830    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x"
Oct 30 05:41:04.740243 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:04.740214    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x"]
Oct 30 05:41:04.860920 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:04.860845    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/396ecffe-9a19-487d-a9cd-5d20d1b65fcf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x\" (UID: \"396ecffe-9a19-487d-a9cd-5d20d1b65fcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x"
Oct 30 05:41:04.860920 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:04.860927    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/396ecffe-9a19-487d-a9cd-5d20d1b65fcf-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x\" (UID: \"396ecffe-9a19-487d-a9cd-5d20d1b65fcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x"
Oct 30 05:41:04.861158 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:04.861019    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/396ecffe-9a19-487d-a9cd-5d20d1b65fcf-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x\" (UID: \"396ecffe-9a19-487d-a9cd-5d20d1b65fcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x"
Oct 30 05:41:04.861158 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:04.861062    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/396ecffe-9a19-487d-a9cd-5d20d1b65fcf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x\" (UID: \"396ecffe-9a19-487d-a9cd-5d20d1b65fcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x"
Oct 30 05:41:04.861158 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:04.861130    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/396ecffe-9a19-487d-a9cd-5d20d1b65fcf-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x\" (UID: \"396ecffe-9a19-487d-a9cd-5d20d1b65fcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x"
Oct 30 05:41:04.861264 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:04.861169    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/396ecffe-9a19-487d-a9cd-5d20d1b65fcf-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x\" (UID: \"396ecffe-9a19-487d-a9cd-5d20d1b65fcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x"
Oct 30 05:41:04.861264 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:04.861196    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/396ecffe-9a19-487d-a9cd-5d20d1b65fcf-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x\" (UID: \"396ecffe-9a19-487d-a9cd-5d20d1b65fcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x"
Oct 30 05:41:04.861334 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:04.861286    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c59k7\" (UniqueName: \"kubernetes.io/projected/396ecffe-9a19-487d-a9cd-5d20d1b65fcf-kube-api-access-c59k7\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x\" (UID: \"396ecffe-9a19-487d-a9cd-5d20d1b65fcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x"
Oct 30 05:41:04.861370 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:04.861333    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/396ecffe-9a19-487d-a9cd-5d20d1b65fcf-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x\" (UID: \"396ecffe-9a19-487d-a9cd-5d20d1b65fcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x"
Oct 30 05:41:04.962570 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:04.962486    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/396ecffe-9a19-487d-a9cd-5d20d1b65fcf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x\" (UID: \"396ecffe-9a19-487d-a9cd-5d20d1b65fcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x"
Oct 30 05:41:04.962570 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:04.962521    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/396ecffe-9a19-487d-a9cd-5d20d1b65fcf-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x\" (UID: \"396ecffe-9a19-487d-a9cd-5d20d1b65fcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x"
Oct 30 05:41:04.962812 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:04.962682    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/396ecffe-9a19-487d-a9cd-5d20d1b65fcf-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x\" (UID: \"396ecffe-9a19-487d-a9cd-5d20d1b65fcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x"
Oct 30 05:41:04.962812 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:04.962725    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/396ecffe-9a19-487d-a9cd-5d20d1b65fcf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x\" (UID: \"396ecffe-9a19-487d-a9cd-5d20d1b65fcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x"
Oct 30 05:41:04.962812 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:04.962751    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/396ecffe-9a19-487d-a9cd-5d20d1b65fcf-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x\" (UID: \"396ecffe-9a19-487d-a9cd-5d20d1b65fcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x"
Oct 30 05:41:04.962812 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:04.962778    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/396ecffe-9a19-487d-a9cd-5d20d1b65fcf-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x\" (UID: \"396ecffe-9a19-487d-a9cd-5d20d1b65fcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x"
Oct 30 05:41:04.962812 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:04.962813    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/396ecffe-9a19-487d-a9cd-5d20d1b65fcf-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x\" (UID: \"396ecffe-9a19-487d-a9cd-5d20d1b65fcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x"
Oct 30 05:41:04.963145 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:04.962851    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c59k7\" (UniqueName: \"kubernetes.io/projected/396ecffe-9a19-487d-a9cd-5d20d1b65fcf-kube-api-access-c59k7\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x\" (UID: \"396ecffe-9a19-487d-a9cd-5d20d1b65fcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x"
Oct 30 05:41:04.963145 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:04.962941    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/396ecffe-9a19-487d-a9cd-5d20d1b65fcf-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x\" (UID: \"396ecffe-9a19-487d-a9cd-5d20d1b65fcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x"
Oct 30 05:41:04.963145 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:04.963049    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/396ecffe-9a19-487d-a9cd-5d20d1b65fcf-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x\" (UID: \"396ecffe-9a19-487d-a9cd-5d20d1b65fcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x"
Oct 30 05:41:04.963145 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:04.963097    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/396ecffe-9a19-487d-a9cd-5d20d1b65fcf-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x\" (UID: \"396ecffe-9a19-487d-a9cd-5d20d1b65fcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x"
Oct 30 05:41:04.963461 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:04.963436    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/396ecffe-9a19-487d-a9cd-5d20d1b65fcf-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x\" (UID: \"396ecffe-9a19-487d-a9cd-5d20d1b65fcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x"
Oct 30 05:41:04.963547 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:04.963497    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/396ecffe-9a19-487d-a9cd-5d20d1b65fcf-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x\" (UID: \"396ecffe-9a19-487d-a9cd-5d20d1b65fcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x"
Oct 30 05:41:04.963618 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:04.963586    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/396ecffe-9a19-487d-a9cd-5d20d1b65fcf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x\" (UID: \"396ecffe-9a19-487d-a9cd-5d20d1b65fcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x"
Oct 30 05:41:04.964977 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:04.964951    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/396ecffe-9a19-487d-a9cd-5d20d1b65fcf-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x\" (UID: \"396ecffe-9a19-487d-a9cd-5d20d1b65fcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x"
Oct 30 05:41:04.965377 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:04.965357    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/396ecffe-9a19-487d-a9cd-5d20d1b65fcf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x\" (UID: \"396ecffe-9a19-487d-a9cd-5d20d1b65fcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x"
Oct 30 05:41:04.982738 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:04.982709    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/396ecffe-9a19-487d-a9cd-5d20d1b65fcf-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x\" (UID: \"396ecffe-9a19-487d-a9cd-5d20d1b65fcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x"
Oct 30 05:41:04.983125 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:04.983106    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c59k7\" (UniqueName: \"kubernetes.io/projected/396ecffe-9a19-487d-a9cd-5d20d1b65fcf-kube-api-access-c59k7\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x\" (UID: \"396ecffe-9a19-487d-a9cd-5d20d1b65fcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x"
Oct 30 05:41:05.023720 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:05.023695    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x"
Oct 30 05:41:05.337168 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:05.337129    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x"]
Oct 30 05:41:05.342336 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:41:05.342311    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod396ecffe_9a19_487d_a9cd_5d20d1b65fcf.slice/crio-364b654c7ad7f7936091ec958000df16a1e4a0c512723e87d00b27aa09b7af05 WatchSource:0}: Error finding container 364b654c7ad7f7936091ec958000df16a1e4a0c512723e87d00b27aa09b7af05: Status 404 returned error can't find the container with id 364b654c7ad7f7936091ec958000df16a1e4a0c512723e87d00b27aa09b7af05
Oct 30 05:41:05.344465 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:05.344436    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:41:05.344539 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:05.344499    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:41:05.344539 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:05.344527    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:41:05.980935 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:05.980201    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x" event={"ID":"396ecffe-9a19-487d-a9cd-5d20d1b65fcf","Type":"ContainerStarted","Data":"78f5bc43b6ce93f995f5843da687a32675b3c132ce8d29de2929c69423a53920"}
Oct 30 05:41:05.980935 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:05.980268    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x" event={"ID":"396ecffe-9a19-487d-a9cd-5d20d1b65fcf","Type":"ContainerStarted","Data":"364b654c7ad7f7936091ec958000df16a1e4a0c512723e87d00b27aa09b7af05"}
Oct 30 05:41:06.024273 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:06.024235    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x"
Oct 30 05:41:06.025694 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:06.025668    5898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 30 05:41:06.025821 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:06.025731    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x" podUID="396ecffe-9a19-487d-a9cd-5d20d1b65fcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 30 05:41:06.045059 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:06.045012    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x" podStartSLOduration=2.044998664 podStartE2EDuration="2.044998664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:41:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:41:06.041793119 +0000 UTC m=+3395.265613041" watchObservedRunningTime="2026-10-30 05:41:06.044998664 +0000 UTC m=+3395.268818564"
Oct 30 05:41:07.024728 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:07.024695    5898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 30 05:41:07.025131 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:07.024745    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x" podUID="396ecffe-9a19-487d-a9cd-5d20d1b65fcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 30 05:41:08.024863 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:08.024830    5898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 30 05:41:08.025282 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:08.024882    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x" podUID="396ecffe-9a19-487d-a9cd-5d20d1b65fcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 30 05:41:09.024568 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:09.024534    5898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 30 05:41:09.024771 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:09.024587    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x" podUID="396ecffe-9a19-487d-a9cd-5d20d1b65fcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 30 05:41:10.024495 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:10.024462    5898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 30 05:41:10.025022 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:10.024517    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x" podUID="396ecffe-9a19-487d-a9cd-5d20d1b65fcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 30 05:41:11.025036 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:11.025000    5898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 30 05:41:11.025440 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:11.025051    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x" podUID="396ecffe-9a19-487d-a9cd-5d20d1b65fcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 30 05:41:12.024315 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:12.024282    5898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 30 05:41:12.024498 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:12.024333    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x" podUID="396ecffe-9a19-487d-a9cd-5d20d1b65fcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 30 05:41:13.567041 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:13.567004    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x"
Oct 30 05:41:13.567545 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:13.567396    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x"
Oct 30 05:41:13.568383 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:13.568362    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x"
Oct 30 05:41:25.947454 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:25.947419    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:41:26.817847 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:41:26.817809    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:41:28.234637 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:28.234570    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:41:44.421179 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:44.421091    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:41:54.505521 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:41:54.505475    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:41:56.830185 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:41:56.830103    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:42:13.581114 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:42:13.581076    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:42:17.715116 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:42:17.715085    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:42:19.932297 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:42:19.932253    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb"]
Oct 30 05:42:19.944108 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:42:19.944082    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb"
Oct 30 05:42:19.958689 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:42:19.958658    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb"]
Oct 30 05:42:19.995654 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:42:19.995624    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:42:20.015736 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:42:20.015699    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b40b768-9853-4acd-bf54-9d90b9f5a0ab-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb\" (UID: \"0b40b768-9853-4acd-bf54-9d90b9f5a0ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb"
Oct 30 05:42:20.015933 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:42:20.015744    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b40b768-9853-4acd-bf54-9d90b9f5a0ab-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb\" (UID: \"0b40b768-9853-4acd-bf54-9d90b9f5a0ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb"
Oct 30 05:42:20.015933 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:42:20.015786    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj4jp\" (UniqueName: \"kubernetes.io/projected/0b40b768-9853-4acd-bf54-9d90b9f5a0ab-kube-api-access-pj4jp\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb\" (UID: \"0b40b768-9853-4acd-bf54-9d90b9f5a0ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb"
Oct 30 05:42:20.015933 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:42:20.015827    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b40b768-9853-4acd-bf54-9d90b9f5a0ab-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb\" (UID: \"0b40b768-9853-4acd-bf54-9d90b9f5a0ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb"
Oct 30 05:42:20.015933 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:42:20.015881    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b40b768-9853-4acd-bf54-9d90b9f5a0ab-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb\" (UID: \"0b40b768-9853-4acd-bf54-9d90b9f5a0ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb"
Oct 30 05:42:20.015933 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:42:20.015931    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b40b768-9853-4acd-bf54-9d90b9f5a0ab-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb\" (UID: \"0b40b768-9853-4acd-bf54-9d90b9f5a0ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb"
Oct 30 05:42:20.016263 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:42:20.015984    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b40b768-9853-4acd-bf54-9d90b9f5a0ab-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb\" (UID: \"0b40b768-9853-4acd-bf54-9d90b9f5a0ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb"
Oct 30 05:42:20.016263 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:42:20.016018    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b40b768-9853-4acd-bf54-9d90b9f5a0ab-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb\" (UID: \"0b40b768-9853-4acd-bf54-9d90b9f5a0ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb"
Oct 30 05:42:20.016263 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:42:20.016060    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b40b768-9853-4acd-bf54-9d90b9f5a0ab-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb\" (UID: \"0b40b768-9853-4acd-bf54-9d90b9f5a0ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb"
Oct 30 05:42:20.117117 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:42:20.117080    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b40b768-9853-4acd-bf54-9d90b9f5a0ab-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb\" (UID: \"0b40b768-9853-4acd-bf54-9d90b9f5a0ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb"
Oct 30 05:42:20.117330 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:42:20.117146    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b40b768-9853-4acd-bf54-9d90b9f5a0ab-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb\" (UID: \"0b40b768-9853-4acd-bf54-9d90b9f5a0ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb"
Oct 30 05:42:20.117330 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:42:20.117198    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b40b768-9853-4acd-bf54-9d90b9f5a0ab-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb\" (UID: \"0b40b768-9853-4acd-bf54-9d90b9f5a0ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb"
Oct 30 05:42:20.117947 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:42:20.117913    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b40b768-9853-4acd-bf54-9d90b9f5a0ab-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb\" (UID: \"0b40b768-9853-4acd-bf54-9d90b9f5a0ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb"
Oct 30 05:42:20.118115 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:42:20.118087    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b40b768-9853-4acd-bf54-9d90b9f5a0ab-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb\" (UID: \"0b40b768-9853-4acd-bf54-9d90b9f5a0ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb"
Oct 30 05:42:20.118348 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:42:20.118322    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b40b768-9853-4acd-bf54-9d90b9f5a0ab-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb\" (UID: \"0b40b768-9853-4acd-bf54-9d90b9f5a0ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb"
Oct 30 05:42:20.118438 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:42:20.118400    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj4jp\" (UniqueName: \"kubernetes.io/projected/0b40b768-9853-4acd-bf54-9d90b9f5a0ab-kube-api-access-pj4jp\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb\" (UID: \"0b40b768-9853-4acd-bf54-9d90b9f5a0ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb"
Oct 30 05:42:20.118741 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:42:20.118719    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b40b768-9853-4acd-bf54-9d90b9f5a0ab-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb\" (UID: \"0b40b768-9853-4acd-bf54-9d90b9f5a0ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb"
Oct 30 05:42:20.118847 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:42:20.118794    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b40b768-9853-4acd-bf54-9d90b9f5a0ab-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb\" (UID: \"0b40b768-9853-4acd-bf54-9d90b9f5a0ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb"
Oct 30 05:42:20.118847 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:42:20.118834    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b40b768-9853-4acd-bf54-9d90b9f5a0ab-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb\" (UID: \"0b40b768-9853-4acd-bf54-9d90b9f5a0ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb"
Oct 30 05:42:20.119215 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:42:20.119191    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b40b768-9853-4acd-bf54-9d90b9f5a0ab-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb\" (UID: \"0b40b768-9853-4acd-bf54-9d90b9f5a0ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb"
Oct 30 05:42:20.119375 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:42:20.119214    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b40b768-9853-4acd-bf54-9d90b9f5a0ab-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb\" (UID: \"0b40b768-9853-4acd-bf54-9d90b9f5a0ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb"
Oct 30 05:42:20.119520 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:42:20.119281    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b40b768-9853-4acd-bf54-9d90b9f5a0ab-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb\" (UID: \"0b40b768-9853-4acd-bf54-9d90b9f5a0ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb"
Oct 30 05:42:20.119737 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:42:20.119608    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b40b768-9853-4acd-bf54-9d90b9f5a0ab-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb\" (UID: \"0b40b768-9853-4acd-bf54-9d90b9f5a0ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb"
Oct 30 05:42:20.121698 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:42:20.121670    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b40b768-9853-4acd-bf54-9d90b9f5a0ab-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb\" (UID: \"0b40b768-9853-4acd-bf54-9d90b9f5a0ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb"
Oct 30 05:42:20.121915 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:42:20.121887    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b40b768-9853-4acd-bf54-9d90b9f5a0ab-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb\" (UID: \"0b40b768-9853-4acd-bf54-9d90b9f5a0ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb"
Oct 30 05:42:20.126006 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:42:20.125939    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:42:20.136197 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:42:20.136176    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b40b768-9853-4acd-bf54-9d90b9f5a0ab-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb\" (UID: \"0b40b768-9853-4acd-bf54-9d90b9f5a0ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb"
Oct 30 05:42:20.137070 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:42:20.137048    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj4jp\" (UniqueName: \"kubernetes.io/projected/0b40b768-9853-4acd-bf54-9d90b9f5a0ab-kube-api-access-pj4jp\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb\" (UID: \"0b40b768-9853-4acd-bf54-9d90b9f5a0ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb"
Oct 30 05:42:20.258427 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:42:20.258334    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb"
Oct 30 05:42:20.585048 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:42:20.585016    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb"]
Oct 30 05:42:20.589679 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:42:20.589647    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b40b768_9853_4acd_bf54_9d90b9f5a0ab.slice/crio-54eee6cd7e8ae3b864d819f6f8ba8dac507dfc67b5612ffeeb4145b02cdcb888 WatchSource:0}: Error finding container 54eee6cd7e8ae3b864d819f6f8ba8dac507dfc67b5612ffeeb4145b02cdcb888: Status 404 returned error can't find the container with id 54eee6cd7e8ae3b864d819f6f8ba8dac507dfc67b5612ffeeb4145b02cdcb888
Oct 30 05:42:20.591744 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:42:20.591711    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:42:20.591813 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:42:20.591782    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:42:20.591855 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:42:20.591812    5898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 05:42:21.373074 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:42:21.373028    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb" event={"ID":"0b40b768-9853-4acd-bf54-9d90b9f5a0ab","Type":"ContainerStarted","Data":"4c5e44457966b25469f39435dd617b33cf9d50d3fe3808eb77be825525beb3be"}
Oct 30 05:42:21.373509 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:42:21.373089    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb" event={"ID":"0b40b768-9853-4acd-bf54-9d90b9f5a0ab","Type":"ContainerStarted","Data":"54eee6cd7e8ae3b864d819f6f8ba8dac507dfc67b5612ffeeb4145b02cdcb888"}
Oct 30 05:42:21.448231 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:42:21.448181    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb" podStartSLOduration=2.4481680089999998 podStartE2EDuration="2.448168009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:42:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:42:21.436559209 +0000 UTC m=+3470.660379110" watchObservedRunningTime="2026-10-30 05:42:21.448168009 +0000 UTC m=+3470.671987910"
Oct 30 05:42:22.258701 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:42:22.258662    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb"
Oct 30 05:42:22.260197 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:42:22.260168    5898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Oct 30 05:42:22.260310 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:42:22.260260    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb" podUID="0b40b768-9853-4acd-bf54-9d90b9f5a0ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Oct 30 05:42:23.259278 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:42:23.259241    5898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Oct 30 05:42:23.259694 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:42:23.259294    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb" podUID="0b40b768-9853-4acd-bf54-9d90b9f5a0ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Oct 30 05:42:24.259515 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:42:24.259478    5898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Oct 30 05:42:24.259978 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:42:24.259536    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb" podUID="0b40b768-9853-4acd-bf54-9d90b9f5a0ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Oct 30 05:42:25.259394 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:42:25.259359    5898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Oct 30 05:42:25.259599 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:42:25.259413    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb" podUID="0b40b768-9853-4acd-bf54-9d90b9f5a0ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Oct 30 05:42:26.259104 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:42:26.259065    5898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Oct 30 05:42:26.259299 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:42:26.259152    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb" podUID="0b40b768-9853-4acd-bf54-9d90b9f5a0ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Oct 30 05:42:26.818040 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:42:26.817998    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:42:27.258971 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:42:27.258936    5898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Oct 30 05:42:27.259213 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:42:27.258992    5898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb" podUID="0b40b768-9853-4acd-bf54-9d90b9f5a0ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Oct 30 05:42:28.772742 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:42:28.772703    5898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb"
Oct 30 05:42:28.773262 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:42:28.773096    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb"
Oct 30 05:42:28.774013 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:42:28.773987    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb"
Oct 30 05:42:33.976522 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:42:33.976488    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:42:44.496387 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:42:44.496350    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:42:49.015283 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:42:49.015246    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:42:49.115952 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:42:49.115917    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:42:49.690398 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:42:49.690370    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:42:56.817628 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:42:56.817583    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:43:13.886753 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:43:13.886661    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:43:22.928279 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:43:22.928249    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:43:24.211459 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:43:24.211419    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:43:26.816975 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:43:26.816940    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:43:27.923570 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:43:27.923540    5898 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/NetworkManager-dispatcher.service\": RecentStats: unable to find data in memory cache]"
Oct 30 05:43:28.682216 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:43:28.682182    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:43:44.614916 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:43:44.614874    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:43:46.365500 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:43:46.365459    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:43:51.036273 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:43:51.036232    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:43:52.692248 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:43:52.692212    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:43:56.817248 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:43:56.817207    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:44:05.080779 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:44:05.080733    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:44:09.817437 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:44:09.817399    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:44:26.818419 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:44:26.818376    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:44:27.109761 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:44:27.109727    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:44:29.609079 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:44:29.609029    5898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 05:44:29.609842 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:44:29.609807    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x" podUID="396ecffe-9a19-487d-a9cd-5d20d1b65fcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:44:29.816056 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:44:29.816010    5898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 05:44:29.816412 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:44:29.816376    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb" podUID="0b40b768-9853-4acd-bf54-9d90b9f5a0ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:44:40.585099 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:44:40.585066    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:44.202521 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:44:44.202412    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:44:44.742672 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:44:44.742636    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:44:54.893539 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:44:54.893173    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:44:56.860837 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:44:56.860790    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:44:59.853759 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:44:59.853723    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:45:03.033279 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:45:03.033247    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-157dec48-qpbkd"]
Oct 30 05:45:03.044372 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:45:03.044343    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-157dec48-qpbkd"]
Oct 30 05:45:04.095442 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:45:04.095393    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6xtvh"]
Oct 30 05:45:04.099196 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:45:04.099173    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6xtvh"
Oct 30 05:45:04.115836 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:45:04.115815    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6xtvh"]
Oct 30 05:45:04.124673 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:45:04.124647    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:45:04.124790 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:45:04.124672    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 05:45:04.124790 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:45:04.124737    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\""
Oct 30 05:45:04.124927 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:45:04.124801    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 05:45:04.199885 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:45:04.199856    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f2602f9-6272-4eca-a004-82facb9469da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-6xtvh\" (UID: \"0f2602f9-6272-4eca-a004-82facb9469da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6xtvh"
Oct 30 05:45:04.200041 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:45:04.199923    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f2602f9-6272-4eca-a004-82facb9469da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-6xtvh\" (UID: \"0f2602f9-6272-4eca-a004-82facb9469da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6xtvh"
Oct 30 05:45:04.200041 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:45:04.199987    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8fdm\" (UniqueName: \"kubernetes.io/projected/0f2602f9-6272-4eca-a004-82facb9469da-kube-api-access-k8fdm\") pod \"maas-api-revoke-sub-3d4156f1-6xtvh\" (UID: \"0f2602f9-6272-4eca-a004-82facb9469da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6xtvh"
Oct 30 05:45:04.300576 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:45:04.300550    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8fdm\" (UniqueName: \"kubernetes.io/projected/0f2602f9-6272-4eca-a004-82facb9469da-kube-api-access-k8fdm\") pod \"maas-api-revoke-sub-3d4156f1-6xtvh\" (UID: \"0f2602f9-6272-4eca-a004-82facb9469da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6xtvh"
Oct 30 05:45:04.300751 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:45:04.300694    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f2602f9-6272-4eca-a004-82facb9469da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-6xtvh\" (UID: \"0f2602f9-6272-4eca-a004-82facb9469da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6xtvh"
Oct 30 05:45:04.300751 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:45:04.300732    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f2602f9-6272-4eca-a004-82facb9469da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-6xtvh\" (UID: \"0f2602f9-6272-4eca-a004-82facb9469da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6xtvh"
Oct 30 05:45:04.301484 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:45:04.301461    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f2602f9-6272-4eca-a004-82facb9469da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-6xtvh\" (UID: \"0f2602f9-6272-4eca-a004-82facb9469da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6xtvh"
Oct 30 05:45:04.303066 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:45:04.303044    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f2602f9-6272-4eca-a004-82facb9469da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-6xtvh\" (UID: \"0f2602f9-6272-4eca-a004-82facb9469da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6xtvh"
Oct 30 05:45:04.318646 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:45:04.318627    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8fdm\" (UniqueName: \"kubernetes.io/projected/0f2602f9-6272-4eca-a004-82facb9469da-kube-api-access-k8fdm\") pod \"maas-api-revoke-sub-3d4156f1-6xtvh\" (UID: \"0f2602f9-6272-4eca-a004-82facb9469da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6xtvh"
Oct 30 05:45:04.412127 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:45:04.412056    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6xtvh"
Oct 30 05:45:04.706071 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:45:04.705979    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6xtvh"]
Oct 30 05:45:04.710631 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:45:04.710606    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f2602f9_6272_4eca_a004_82facb9469da.slice/crio-1070a03c6319faf1b353062d5a9e33fd9036f1e69ff50199e3a741ff5e557499 WatchSource:0}: Error finding container 1070a03c6319faf1b353062d5a9e33fd9036f1e69ff50199e3a741ff5e557499: Status 404 returned error can't find the container with id 1070a03c6319faf1b353062d5a9e33fd9036f1e69ff50199e3a741ff5e557499
Oct 30 05:45:04.888244 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:45:04.888213    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="274129ce-5c11-4060-ab39-856dd097ffb1" path="/var/lib/kubelet/pods/274129ce-5c11-4060-ab39-856dd097ffb1/volumes"
Oct 30 05:45:05.303449 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:45:05.303362    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6xtvh" event={"ID":"0f2602f9-6272-4eca-a004-82facb9469da","Type":"ContainerStarted","Data":"b99b8026177fecbd589ad0f4b22d4907bbe84a4515a101238b9d94dbcc38bdd2"}
Oct 30 05:45:05.303449 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:45:05.303412    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6xtvh" event={"ID":"0f2602f9-6272-4eca-a004-82facb9469da","Type":"ContainerStarted","Data":"1070a03c6319faf1b353062d5a9e33fd9036f1e69ff50199e3a741ff5e557499"}
Oct 30 05:45:05.341618 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:45:05.341563    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6xtvh" podStartSLOduration=1.341544717 podStartE2EDuration="1.341544717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:45:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:45:05.339107576 +0000 UTC m=+3634.562927478" watchObservedRunningTime="2026-10-30 05:45:05.341544717 +0000 UTC m=+3634.565364619"
Oct 30 05:45:06.310487 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:45:06.310455    5898 generic.go:356] "Generic (PLEG): container finished" podID="0f2602f9-6272-4eca-a004-82facb9469da" containerID="b99b8026177fecbd589ad0f4b22d4907bbe84a4515a101238b9d94dbcc38bdd2" exitCode=0
Oct 30 05:45:06.310917 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:45:06.310541    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6xtvh" event={"ID":"0f2602f9-6272-4eca-a004-82facb9469da","Type":"ContainerDied","Data":"b99b8026177fecbd589ad0f4b22d4907bbe84a4515a101238b9d94dbcc38bdd2"}
Oct 30 05:45:07.440984 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:45:07.440961    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6xtvh"
Oct 30 05:45:07.631841 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:45:07.631806    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f2602f9-6272-4eca-a004-82facb9469da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f2602f9-6272-4eca-a004-82facb9469da-maas-api-xks-ca\") pod \"0f2602f9-6272-4eca-a004-82facb9469da\" (UID: \"0f2602f9-6272-4eca-a004-82facb9469da\") "
Oct 30 05:45:07.632029 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:45:07.631886    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f2602f9-6272-4eca-a004-82facb9469da-kube-api-access-k8fdm\" (UniqueName: \"kubernetes.io/projected/0f2602f9-6272-4eca-a004-82facb9469da-kube-api-access-k8fdm\") pod \"0f2602f9-6272-4eca-a004-82facb9469da\" (UID: \"0f2602f9-6272-4eca-a004-82facb9469da\") "
Oct 30 05:45:07.632029 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:45:07.631948    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f2602f9-6272-4eca-a004-82facb9469da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f2602f9-6272-4eca-a004-82facb9469da-maas-api-openshift-ca\") pod \"0f2602f9-6272-4eca-a004-82facb9469da\" (UID: \"0f2602f9-6272-4eca-a004-82facb9469da\") "
Oct 30 05:45:07.632345 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:45:07.632324    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f2602f9-6272-4eca-a004-82facb9469da-maas-api-openshift-ca" pod "0f2602f9-6272-4eca-a004-82facb9469da" (UID: "0f2602f9-6272-4eca-a004-82facb9469da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:45:07.633994 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:45:07.633975    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f2602f9-6272-4eca-a004-82facb9469da-maas-api-xks-ca" pod "0f2602f9-6272-4eca-a004-82facb9469da" (UID: "0f2602f9-6272-4eca-a004-82facb9469da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:45:07.634092 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:45:07.634034    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f2602f9-6272-4eca-a004-82facb9469da-kube-api-access-k8fdm" pod "0f2602f9-6272-4eca-a004-82facb9469da" (UID: "0f2602f9-6272-4eca-a004-82facb9469da"). InnerVolumeSpecName "kube-api-access-k8fdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:45:07.733420 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:45:07.733385    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f2602f9-6272-4eca-a004-82facb9469da-maas-api-xks-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:45:07.733420 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:45:07.733415    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8fdm\" (UniqueName: \"kubernetes.io/projected/0f2602f9-6272-4eca-a004-82facb9469da-kube-api-access-k8fdm\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:45:07.733420 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:45:07.733425    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f2602f9-6272-4eca-a004-82facb9469da-maas-api-openshift-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:45:08.320584 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:45:08.320558    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6xtvh"
Oct 30 05:45:08.320584 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:45:08.320570    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6xtvh" event={"ID":"0f2602f9-6272-4eca-a004-82facb9469da","Type":"ContainerDied","Data":"1070a03c6319faf1b353062d5a9e33fd9036f1e69ff50199e3a741ff5e557499"}
Oct 30 05:45:08.320864 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:45:08.320635    5898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1070a03c6319faf1b353062d5a9e33fd9036f1e69ff50199e3a741ff5e557499"
Oct 30 05:45:15.894380 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:45:15.894335    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:45:20.598559 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:45:20.598510    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:45:25.025390 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:45:25.025342    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1375b879-tnj9g"]
Oct 30 05:45:25.036490 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:45:25.036463    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1375b879-tnj9g"]
Oct 30 05:45:26.703071 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:45:26.703035    5898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6xtvh"]
Oct 30 05:45:26.714003 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:45:26.713978    5898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6xtvh"]
Oct 30 05:45:26.817541 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:45:26.817498    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:45:26.888086 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:45:26.888054    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f2602f9-6272-4eca-a004-82facb9469da" path="/var/lib/kubelet/pods/0f2602f9-6272-4eca-a004-82facb9469da/volumes"
Oct 30 05:45:26.888566 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:45:26.888551    5898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19904b19-a70f-4ec6-b99e-94b153f58f67" path="/var/lib/kubelet/pods/19904b19-a70f-4ec6-b99e-94b153f58f67/volumes"
Oct 30 05:45:41.830606 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:45:41.830569    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:45:42.304544 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:45:42.304458    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:45:42.402162 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:45:42.402127    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:45:44.850573 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:45:44.850539    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:45:46.151095 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:45:46.151060    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:45:46.268519 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:45:46.268484    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:45:47.696981 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:45:47.696939    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:45:47.797377 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:45:47.797338    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:45:47.909670 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:45:47.909635    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:45:48.016333 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:45:48.016247    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:45:49.661758 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:45:49.661724    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:45:51.284588 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:45:51.284554    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:45:51.357301 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:45:51.357258    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:45:53.269286 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:45:53.269226    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:45:55.129669 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:45:55.129635    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:45:56.818031 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:45:56.817991    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:45:57.089318 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:45:57.089279    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:45:58.887109 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:45:58.887058    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:00.709454 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:46:00.709422    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:01.184135 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:46:01.184093    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:01.758474 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:46:01.758414    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:01.931607 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:46:01.931550    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:02.920817 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:46:02.920768    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:04.285345 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:46:04.285314    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:04.873650 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:46:04.873617    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:06.298210 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:46:06.298174    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:46:06.791313 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:46:06.791274    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:08.667343 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:46:08.667301    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:10.519465 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:46:10.519428    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:12.272322 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:46:12.272245    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:14.026783 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:46:14.026750    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:15.782351 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:46:15.782317    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:17.604184 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:46:17.604149    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:19.486968 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:46:19.486930    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:21.123281 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:46:21.123247    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:46:21.506840 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:46:21.506721    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:23.351060 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:46:23.351027    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:25.005243 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:46:25.005213    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:26.691287 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:46:26.691253    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:26.817174 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:46:26.817135    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:46:28.656789 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:46:28.656756    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:30.386474 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:46:30.386437    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:31.543552 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:46:31.543515    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:32.240024 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:46:32.239989    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:33.650085 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:46:33.650053    5898 scope.go:122] "RemoveContainer" containerID="24bede84efbc610521ee26d72b14c7da371b048cf8c5e17278439fdca0377154"
Oct 30 05:46:33.658519 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:46:33.658501    5898 scope.go:122] "RemoveContainer" containerID="a18666e716209aab884c1d15d6ffd1943afe544a337cce54d82656c739de6657"
Oct 30 05:46:33.909613 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:46:33.909525    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:35.566479 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:46:35.566444    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:37.370800 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:46:37.370761    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:38.261829 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:46:38.261785    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:38.498423 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:46:38.498388    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:46:39.268987 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:46:39.268955    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:41.079115 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:46:41.079087    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:42.817218 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:46:42.817182    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:43.485454 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:46:43.485418    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:44.649544 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:46:44.649515    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:44.974607 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:46:44.974524    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:46.467952 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:46:46.467891    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:48.155034 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:46:48.154999    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:49.139849 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:46:49.139816    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:49.255386 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:46:49.255340    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:49.662150 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:46:49.662114    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:51.725513 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:46:51.725477    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:46:52.308142 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:46:52.308109    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:52.452008 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:46:52.451950    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:52.950674 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:46:52.950590    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:56.817395 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:46:56.817351    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:46:59.576720 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:46:59.576691    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:59.693750 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:46:59.693711    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:46:59.958794 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:46:59.958699    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:47:15.684996 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:47:15.684961    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:47:20.328528 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:47:20.328495    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:47:20.724971 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:47:20.724869    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:47:26.312012 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:47:26.311974    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:47:26.817797 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:47:26.817759    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:47:45.092241 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:47:45.092152    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:47:56.817069 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:47:56.817029    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:48:26.817774 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:48:26.817741    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:48:42.284783 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:48:42.284744    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:48:44.615144 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:48:44.614991    5898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 05:48:44.620043 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:48:44.615659    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x" podUID="396ecffe-9a19-487d-a9cd-5d20d1b65fcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:48:44.822081 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:48:44.822030    5898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 05:48:44.822430 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:48:44.822378    5898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb" podUID="0b40b768-9853-4acd-bf54-9d90b9f5a0ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:48:45.286482 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:48:45.286449    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:48:56.817680 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:48:56.817644    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:48:57.473961 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:48:57.473924    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:49:14.668222 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:49:14.668130    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:49:15.515104 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:49:15.515064    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:49:18.253639 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:49:18.253596    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:49:18.356966 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:49:18.356930    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:49:18.458774 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:49:18.458727    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:49:20.028176 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:49:20.028091    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:49:20.126398 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:49:20.126362    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:49:20.226159 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:49:20.226122    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:49:21.819950 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:49:21.819920    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:49:23.364117 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:49:23.364082    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:49:23.708687 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:49:23.708574    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:49:25.429696 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:49:25.429653    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:49:26.816777 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:49:26.816739    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:49:27.160316 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:49:27.160266    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:49:28.854823 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:49:28.854786    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:49:30.757073 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:49:30.757042    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:49:32.525119 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:49:32.525065    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:49:33.213106 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:49:33.213021    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:49:33.807800 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:49:33.807768    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:49:33.906193 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:49:33.906158    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:49:34.386636 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:49:34.386592    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:49:35.202231 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:49:35.202201    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:49:36.386076 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:49:36.386042    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:49:37.176652 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:49:37.176619    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:49:38.256127 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:49:38.256098    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:49:40.078050 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:49:40.078016    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:49:41.965420 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:49:41.965376    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:49:43.563052 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:49:43.563021    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:49:45.229706 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:49:45.229668    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:49:45.402820 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:49:45.402790    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:49:46.983597 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:49:46.983563    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:49:48.695295 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:49:48.695253    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:49:50.561093 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:49:50.561058    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:49:52.255639 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:49:52.255605    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:49:52.771939 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:49:52.771872    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:49:54.060865 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:49:54.060833    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:49:55.689546 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:49:55.689512    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:49:56.817607 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:49:56.817571    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:49:57.304210 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:49:57.304178    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:49:59.200845 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:49:59.200815    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:00.792058 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:50:00.792025    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:02.506857 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:50:02.506821    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:04.290321 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:50:04.290286    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:05.568084 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:50:05.568050    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:06.008927 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:50:06.008872    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:06.170656 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:50:06.170618    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:50:07.979868 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:50:07.979831    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:09.794205 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:50:09.794163    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:10.657981 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:50:10.657952    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:11.476879 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:50:11.476824    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:12.361734 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:50:12.361687    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:12.465086 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:50:12.465037    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:13.141561 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:50:13.141513    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:14.991663 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:50:14.991625    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:16.417370 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:50:16.417321    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:16.520045 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:50:16.520001    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:17.790485 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:50:17.790451    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:17.889263 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:50:17.889225    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:18.405187 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:50:18.405154    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:19.462168 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:50:19.462134    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:50:22.182371 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:50:22.182334    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:22.262377 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:50:22.262346    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:24.084194 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:50:24.084156    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:24.184489 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:50:24.184454    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:24.455580 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:50:24.455476    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:26.579421 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:50:26.579386    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:26.817599 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:50:26.817563    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:50:29.039087 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:50:29.039051    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:30.892648 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:50:30.892614    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:50:45.519831 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:50:45.519740    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:45.850046 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:50:45.850008    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:47.887032 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:50:47.886998    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:48.046554 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:50:48.046522    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:50:48.780919 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:50:48.780864    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:50:56.816685 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:50:56.816642    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:51:02.462151 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:51:02.462119    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:03.793126 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:51:03.793088    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:51:16.141141 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:51:16.141107    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:17.674838 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:51:17.674809    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:17.976592 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:51:17.976512    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:51:24.485926 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:51:24.483141    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:51:26.818001 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:51:26.817965    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:51:33.264103 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:51:33.264064    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:51:33.726661 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:51:33.726615    5898 scope.go:122] "RemoveContainer" containerID="b99b8026177fecbd589ad0f4b22d4907bbe84a4515a101238b9d94dbcc38bdd2"
Oct 30 05:51:40.164155 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:51:40.164122    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:51:45.673108 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:51:45.673071    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:51:56.676555 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:51:56.676518    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:51:56.817893 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:51:56.817858    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:52:06.572101 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:06.572068    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:52:14.866107 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:14.866025    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:52:21.551765 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:21.551724    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lmtqm"]
Oct 30 05:52:21.552310 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:21.552092    5898 cpu_manager.go:414] "containerMap: removing container" podUID="0f2602f9-6272-4eca-a004-82facb9469da" containerName="revoke-keys"
Oct 30 05:52:21.552310 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:21.552105    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f2602f9-6272-4eca-a004-82facb9469da" containerName="revoke-keys"
Oct 30 05:52:21.552310 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:21.552183    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f2602f9-6272-4eca-a004-82facb9469da" containerName="revoke-keys"
Oct 30 05:52:21.555329 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:21.555312    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lmtqm"
Oct 30 05:52:21.556778 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:21.556755    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkps2\" (UniqueName: \"kubernetes.io/projected/6d0bd3d6-aa1f-40c9-a0ea-e10fe0308a79-kube-api-access-pkps2\") pod \"maas-api-revoke-sub-f03629db-lmtqm\" (UID: \"6d0bd3d6-aa1f-40c9-a0ea-e10fe0308a79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lmtqm"
Oct 30 05:52:21.556949 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:21.556811    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d0bd3d6-aa1f-40c9-a0ea-e10fe0308a79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-lmtqm\" (UID: \"6d0bd3d6-aa1f-40c9-a0ea-e10fe0308a79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lmtqm"
Oct 30 05:52:21.556949 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:21.556887    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d0bd3d6-aa1f-40c9-a0ea-e10fe0308a79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-lmtqm\" (UID: \"6d0bd3d6-aa1f-40c9-a0ea-e10fe0308a79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lmtqm"
Oct 30 05:52:21.569998 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:21.569979    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:52:21.581633 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:21.581608    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 05:52:21.581816 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:21.581664    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\""
Oct 30 05:52:21.581997 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:21.581975    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 05:52:21.583788 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:21.583763    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lmtqm"]
Oct 30 05:52:21.657478 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:21.657438    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkps2\" (UniqueName: \"kubernetes.io/projected/6d0bd3d6-aa1f-40c9-a0ea-e10fe0308a79-kube-api-access-pkps2\") pod \"maas-api-revoke-sub-f03629db-lmtqm\" (UID: \"6d0bd3d6-aa1f-40c9-a0ea-e10fe0308a79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lmtqm"
Oct 30 05:52:21.657670 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:21.657495    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d0bd3d6-aa1f-40c9-a0ea-e10fe0308a79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-lmtqm\" (UID: \"6d0bd3d6-aa1f-40c9-a0ea-e10fe0308a79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lmtqm"
Oct 30 05:52:21.657670 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:21.657511    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d0bd3d6-aa1f-40c9-a0ea-e10fe0308a79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-lmtqm\" (UID: \"6d0bd3d6-aa1f-40c9-a0ea-e10fe0308a79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lmtqm"
Oct 30 05:52:21.658190 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:21.658171    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d0bd3d6-aa1f-40c9-a0ea-e10fe0308a79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-lmtqm\" (UID: \"6d0bd3d6-aa1f-40c9-a0ea-e10fe0308a79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lmtqm"
Oct 30 05:52:21.660029 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:21.660003    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d0bd3d6-aa1f-40c9-a0ea-e10fe0308a79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-lmtqm\" (UID: \"6d0bd3d6-aa1f-40c9-a0ea-e10fe0308a79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lmtqm"
Oct 30 05:52:21.675909 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:21.675863    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkps2\" (UniqueName: \"kubernetes.io/projected/6d0bd3d6-aa1f-40c9-a0ea-e10fe0308a79-kube-api-access-pkps2\") pod \"maas-api-revoke-sub-f03629db-lmtqm\" (UID: \"6d0bd3d6-aa1f-40c9-a0ea-e10fe0308a79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lmtqm"
Oct 30 05:52:21.868634 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:21.868602    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lmtqm"
Oct 30 05:52:22.161890 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:22.161853    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lmtqm"]
Oct 30 05:52:22.164609 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:52:22.164582    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d0bd3d6_aa1f_40c9_a0ea_e10fe0308a79.slice/crio-34a59ccecdf119ec63f556d4bd1d68f0bf6733ad7adb8eed53afb9a5a932f732 WatchSource:0}: Error finding container 34a59ccecdf119ec63f556d4bd1d68f0bf6733ad7adb8eed53afb9a5a932f732: Status 404 returned error can't find the container with id 34a59ccecdf119ec63f556d4bd1d68f0bf6733ad7adb8eed53afb9a5a932f732
Oct 30 05:52:22.598409 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:22.598374    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lmtqm" event={"ID":"6d0bd3d6-aa1f-40c9-a0ea-e10fe0308a79","Type":"ContainerStarted","Data":"9c4b1b7b9f680fefc2b63f9c5e6168f60fe593f54ba9a58303deb9296e5d7dd1"}
Oct 30 05:52:22.598761 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:22.598429    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lmtqm" event={"ID":"6d0bd3d6-aa1f-40c9-a0ea-e10fe0308a79","Type":"ContainerStarted","Data":"34a59ccecdf119ec63f556d4bd1d68f0bf6733ad7adb8eed53afb9a5a932f732"}
Oct 30 05:52:22.637843 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:22.637792    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lmtqm" podStartSLOduration=1.637780211 podStartE2EDuration="1.637780211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:52:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:52:22.637352365 +0000 UTC m=+4071.861172265" watchObservedRunningTime="2026-10-30 05:52:22.637780211 +0000 UTC m=+4071.861600109"
Oct 30 05:52:23.604884 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:23.604850    5898 generic.go:356] "Generic (PLEG): container finished" podID="6d0bd3d6-aa1f-40c9-a0ea-e10fe0308a79" containerID="9c4b1b7b9f680fefc2b63f9c5e6168f60fe593f54ba9a58303deb9296e5d7dd1" exitCode=0
Oct 30 05:52:23.605333 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:23.604938    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lmtqm" event={"ID":"6d0bd3d6-aa1f-40c9-a0ea-e10fe0308a79","Type":"ContainerDied","Data":"9c4b1b7b9f680fefc2b63f9c5e6168f60fe593f54ba9a58303deb9296e5d7dd1"}
Oct 30 05:52:24.745238 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:24.745214    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lmtqm"
Oct 30 05:52:24.780061 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:24.780035    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d0bd3d6-aa1f-40c9-a0ea-e10fe0308a79-kube-api-access-pkps2\" (UniqueName: \"kubernetes.io/projected/6d0bd3d6-aa1f-40c9-a0ea-e10fe0308a79-kube-api-access-pkps2\") pod \"6d0bd3d6-aa1f-40c9-a0ea-e10fe0308a79\" (UID: \"6d0bd3d6-aa1f-40c9-a0ea-e10fe0308a79\") "
Oct 30 05:52:24.780176 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:24.780120    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d0bd3d6-aa1f-40c9-a0ea-e10fe0308a79-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d0bd3d6-aa1f-40c9-a0ea-e10fe0308a79-maas-api-openshift-ca\") pod \"6d0bd3d6-aa1f-40c9-a0ea-e10fe0308a79\" (UID: \"6d0bd3d6-aa1f-40c9-a0ea-e10fe0308a79\") "
Oct 30 05:52:24.780176 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:24.780164    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d0bd3d6-aa1f-40c9-a0ea-e10fe0308a79-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d0bd3d6-aa1f-40c9-a0ea-e10fe0308a79-maas-api-xks-ca\") pod \"6d0bd3d6-aa1f-40c9-a0ea-e10fe0308a79\" (UID: \"6d0bd3d6-aa1f-40c9-a0ea-e10fe0308a79\") "
Oct 30 05:52:24.780454 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:24.780435    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d0bd3d6-aa1f-40c9-a0ea-e10fe0308a79-maas-api-openshift-ca" pod "6d0bd3d6-aa1f-40c9-a0ea-e10fe0308a79" (UID: "6d0bd3d6-aa1f-40c9-a0ea-e10fe0308a79"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:52:24.782288 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:24.782259    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d0bd3d6-aa1f-40c9-a0ea-e10fe0308a79-kube-api-access-pkps2" pod "6d0bd3d6-aa1f-40c9-a0ea-e10fe0308a79" (UID: "6d0bd3d6-aa1f-40c9-a0ea-e10fe0308a79"). InnerVolumeSpecName "kube-api-access-pkps2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:52:24.782425 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:24.782315    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d0bd3d6-aa1f-40c9-a0ea-e10fe0308a79-maas-api-xks-ca" pod "6d0bd3d6-aa1f-40c9-a0ea-e10fe0308a79" (UID: "6d0bd3d6-aa1f-40c9-a0ea-e10fe0308a79"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:52:24.880985 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:24.880892    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkps2\" (UniqueName: \"kubernetes.io/projected/6d0bd3d6-aa1f-40c9-a0ea-e10fe0308a79-kube-api-access-pkps2\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:52:24.880985 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:24.880943    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d0bd3d6-aa1f-40c9-a0ea-e10fe0308a79-maas-api-openshift-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:52:24.880985 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:24.880954    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d0bd3d6-aa1f-40c9-a0ea-e10fe0308a79-maas-api-xks-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:52:25.616269 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:25.616236    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lmtqm"
Oct 30 05:52:25.616269 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:25.616263    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lmtqm" event={"ID":"6d0bd3d6-aa1f-40c9-a0ea-e10fe0308a79","Type":"ContainerDied","Data":"34a59ccecdf119ec63f556d4bd1d68f0bf6733ad7adb8eed53afb9a5a932f732"}
Oct 30 05:52:25.616557 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:25.616305    5898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34a59ccecdf119ec63f556d4bd1d68f0bf6733ad7adb8eed53afb9a5a932f732"
Oct 30 05:52:26.817074 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:52:26.817027    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:52:32.366036 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:32.365998    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:32.686016 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:32.685935    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:52:33.807752 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:33.807721    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:38.231299 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:38.231262    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-cnvz8"]
Oct 30 05:52:38.231747 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:38.231728    5898 cpu_manager.go:414] "containerMap: removing container" podUID="6d0bd3d6-aa1f-40c9-a0ea-e10fe0308a79" containerName="revoke-keys"
Oct 30 05:52:38.231795 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:38.231748    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d0bd3d6-aa1f-40c9-a0ea-e10fe0308a79" containerName="revoke-keys"
Oct 30 05:52:38.231832 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:38.231818    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d0bd3d6-aa1f-40c9-a0ea-e10fe0308a79" containerName="revoke-keys"
Oct 30 05:52:38.234817 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:38.234800    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-cnvz8"
Oct 30 05:52:38.248864 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:38.248842    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:52:38.260632 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:38.260603    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\""
Oct 30 05:52:38.260783 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:38.260683    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 05:52:38.260837 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:38.260792    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 05:52:38.262780 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:38.262756    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-cnvz8"]
Oct 30 05:52:38.340100 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:38.340066    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a336d4a9-4960-473b-ac17-1359040170af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-cnvz8\" (UID: \"a336d4a9-4960-473b-ac17-1359040170af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-cnvz8"
Oct 30 05:52:38.340288 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:38.340152    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a336d4a9-4960-473b-ac17-1359040170af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-cnvz8\" (UID: \"a336d4a9-4960-473b-ac17-1359040170af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-cnvz8"
Oct 30 05:52:38.340288 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:38.340275    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hnbm\" (UniqueName: \"kubernetes.io/projected/a336d4a9-4960-473b-ac17-1359040170af-kube-api-access-2hnbm\") pod \"maas-api-revoke-sub-0165d7d2-cnvz8\" (UID: \"a336d4a9-4960-473b-ac17-1359040170af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-cnvz8"
Oct 30 05:52:38.441287 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:38.441250    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a336d4a9-4960-473b-ac17-1359040170af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-cnvz8\" (UID: \"a336d4a9-4960-473b-ac17-1359040170af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-cnvz8"
Oct 30 05:52:38.441490 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:38.441323    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hnbm\" (UniqueName: \"kubernetes.io/projected/a336d4a9-4960-473b-ac17-1359040170af-kube-api-access-2hnbm\") pod \"maas-api-revoke-sub-0165d7d2-cnvz8\" (UID: \"a336d4a9-4960-473b-ac17-1359040170af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-cnvz8"
Oct 30 05:52:38.441490 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:38.441355    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a336d4a9-4960-473b-ac17-1359040170af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-cnvz8\" (UID: \"a336d4a9-4960-473b-ac17-1359040170af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-cnvz8"
Oct 30 05:52:38.442003 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:38.441983    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a336d4a9-4960-473b-ac17-1359040170af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-cnvz8\" (UID: \"a336d4a9-4960-473b-ac17-1359040170af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-cnvz8"
Oct 30 05:52:38.443646 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:38.443616    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a336d4a9-4960-473b-ac17-1359040170af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-cnvz8\" (UID: \"a336d4a9-4960-473b-ac17-1359040170af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-cnvz8"
Oct 30 05:52:38.460238 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:38.460213    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hnbm\" (UniqueName: \"kubernetes.io/projected/a336d4a9-4960-473b-ac17-1359040170af-kube-api-access-2hnbm\") pod \"maas-api-revoke-sub-0165d7d2-cnvz8\" (UID: \"a336d4a9-4960-473b-ac17-1359040170af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-cnvz8"
Oct 30 05:52:38.551924 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:38.551835    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-cnvz8"
Oct 30 05:52:38.844503 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:38.844479    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-cnvz8"]
Oct 30 05:52:38.846869 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:52:38.846845    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda336d4a9_4960_473b_ac17_1359040170af.slice/crio-afc1dfca6afc3fe05ebf9eea2f21ddb6f8f123f8a6cffa18a18c14f8465b67aa WatchSource:0}: Error finding container afc1dfca6afc3fe05ebf9eea2f21ddb6f8f123f8a6cffa18a18c14f8465b67aa: Status 404 returned error can't find the container with id afc1dfca6afc3fe05ebf9eea2f21ddb6f8f123f8a6cffa18a18c14f8465b67aa
Oct 30 05:52:39.685096 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:39.685063    5898 generic.go:356] "Generic (PLEG): container finished" podID="a336d4a9-4960-473b-ac17-1359040170af" containerID="874b603c9cbc3f444c181f690cba1271ef0f0c4586d2183a0245eb3cbfb3cc32" exitCode=0
Oct 30 05:52:39.685425 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:39.685145    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-cnvz8" event={"ID":"a336d4a9-4960-473b-ac17-1359040170af","Type":"ContainerDied","Data":"874b603c9cbc3f444c181f690cba1271ef0f0c4586d2183a0245eb3cbfb3cc32"}
Oct 30 05:52:39.685425 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:39.685188    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-cnvz8" event={"ID":"a336d4a9-4960-473b-ac17-1359040170af","Type":"ContainerStarted","Data":"afc1dfca6afc3fe05ebf9eea2f21ddb6f8f123f8a6cffa18a18c14f8465b67aa"}
Oct 30 05:52:40.819440 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:40.819417    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-cnvz8"
Oct 30 05:52:40.967088 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:40.967015    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a336d4a9-4960-473b-ac17-1359040170af-kube-api-access-2hnbm\" (UniqueName: \"kubernetes.io/projected/a336d4a9-4960-473b-ac17-1359040170af-kube-api-access-2hnbm\") pod \"a336d4a9-4960-473b-ac17-1359040170af\" (UID: \"a336d4a9-4960-473b-ac17-1359040170af\") "
Oct 30 05:52:40.967088 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:40.967063    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a336d4a9-4960-473b-ac17-1359040170af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a336d4a9-4960-473b-ac17-1359040170af-maas-api-xks-ca\") pod \"a336d4a9-4960-473b-ac17-1359040170af\" (UID: \"a336d4a9-4960-473b-ac17-1359040170af\") "
Oct 30 05:52:40.967312 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:40.967123    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a336d4a9-4960-473b-ac17-1359040170af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a336d4a9-4960-473b-ac17-1359040170af-maas-api-openshift-ca\") pod \"a336d4a9-4960-473b-ac17-1359040170af\" (UID: \"a336d4a9-4960-473b-ac17-1359040170af\") "
Oct 30 05:52:40.967539 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:40.967514    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a336d4a9-4960-473b-ac17-1359040170af-maas-api-openshift-ca" pod "a336d4a9-4960-473b-ac17-1359040170af" (UID: "a336d4a9-4960-473b-ac17-1359040170af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:52:40.969113 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:40.969089    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a336d4a9-4960-473b-ac17-1359040170af-maas-api-xks-ca" pod "a336d4a9-4960-473b-ac17-1359040170af" (UID: "a336d4a9-4960-473b-ac17-1359040170af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:52:40.969205 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:40.969144    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a336d4a9-4960-473b-ac17-1359040170af-kube-api-access-2hnbm" pod "a336d4a9-4960-473b-ac17-1359040170af" (UID: "a336d4a9-4960-473b-ac17-1359040170af"). InnerVolumeSpecName "kube-api-access-2hnbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:52:41.067980 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:41.067953    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a336d4a9-4960-473b-ac17-1359040170af-maas-api-openshift-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:52:41.067980 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:41.067979    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hnbm\" (UniqueName: \"kubernetes.io/projected/a336d4a9-4960-473b-ac17-1359040170af-kube-api-access-2hnbm\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:52:41.068124 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:41.067989    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a336d4a9-4960-473b-ac17-1359040170af-maas-api-xks-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:52:41.696041 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:41.696010    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-cnvz8"
Oct 30 05:52:41.696041 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:41.696021    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-cnvz8" event={"ID":"a336d4a9-4960-473b-ac17-1359040170af","Type":"ContainerDied","Data":"afc1dfca6afc3fe05ebf9eea2f21ddb6f8f123f8a6cffa18a18c14f8465b67aa"}
Oct 30 05:52:41.696261 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:41.696072    5898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afc1dfca6afc3fe05ebf9eea2f21ddb6f8f123f8a6cffa18a18c14f8465b67aa"
Oct 30 05:52:45.754238 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:45.754203    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:49.238078 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:49.238035    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-r827w"]
Oct 30 05:52:49.238665 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:49.238644    5898 cpu_manager.go:414] "containerMap: removing container" podUID="a336d4a9-4960-473b-ac17-1359040170af" containerName="revoke-keys"
Oct 30 05:52:49.238718 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:49.238670    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a336d4a9-4960-473b-ac17-1359040170af" containerName="revoke-keys"
Oct 30 05:52:49.238809 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:49.238795    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="a336d4a9-4960-473b-ac17-1359040170af" containerName="revoke-keys"
Oct 30 05:52:49.244661 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:49.244643    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-r827w"
Oct 30 05:52:49.271120 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:49.271088    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:52:49.271328 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:49.271112    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 05:52:49.271328 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:49.271162    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vm5z6\""
Oct 30 05:52:49.271525 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:49.271187    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 05:52:49.273356 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:49.273330    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-r827w"]
Oct 30 05:52:49.341740 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:49.341709    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf2cb\" (UniqueName: \"kubernetes.io/projected/cc8175e4-b674-4586-8e97-e6fc60ef5480-kube-api-access-gf2cb\") pod \"maas-api-revoke-sub-6b19338f-r827w\" (UID: \"cc8175e4-b674-4586-8e97-e6fc60ef5480\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-r827w"
Oct 30 05:52:49.341740 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:49.341745    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc8175e4-b674-4586-8e97-e6fc60ef5480-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-r827w\" (UID: \"cc8175e4-b674-4586-8e97-e6fc60ef5480\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-r827w"
Oct 30 05:52:49.341963 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:49.341869    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc8175e4-b674-4586-8e97-e6fc60ef5480-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-r827w\" (UID: \"cc8175e4-b674-4586-8e97-e6fc60ef5480\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-r827w"
Oct 30 05:52:49.442328 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:49.442286    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc8175e4-b674-4586-8e97-e6fc60ef5480-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-r827w\" (UID: \"cc8175e4-b674-4586-8e97-e6fc60ef5480\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-r827w"
Oct 30 05:52:49.442529 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:49.442353    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf2cb\" (UniqueName: \"kubernetes.io/projected/cc8175e4-b674-4586-8e97-e6fc60ef5480-kube-api-access-gf2cb\") pod \"maas-api-revoke-sub-6b19338f-r827w\" (UID: \"cc8175e4-b674-4586-8e97-e6fc60ef5480\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-r827w"
Oct 30 05:52:49.442529 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:49.442378    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc8175e4-b674-4586-8e97-e6fc60ef5480-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-r827w\" (UID: \"cc8175e4-b674-4586-8e97-e6fc60ef5480\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-r827w"
Oct 30 05:52:49.442994 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:49.442969    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc8175e4-b674-4586-8e97-e6fc60ef5480-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-r827w\" (UID: \"cc8175e4-b674-4586-8e97-e6fc60ef5480\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-r827w"
Oct 30 05:52:49.444753 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:49.444730    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc8175e4-b674-4586-8e97-e6fc60ef5480-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-r827w\" (UID: \"cc8175e4-b674-4586-8e97-e6fc60ef5480\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-r827w"
Oct 30 05:52:49.461508 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:49.461487    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf2cb\" (UniqueName: \"kubernetes.io/projected/cc8175e4-b674-4586-8e97-e6fc60ef5480-kube-api-access-gf2cb\") pod \"maas-api-revoke-sub-6b19338f-r827w\" (UID: \"cc8175e4-b674-4586-8e97-e6fc60ef5480\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-r827w"
Oct 30 05:52:49.557249 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:49.557178    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-r827w"
Oct 30 05:52:49.867538 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:49.867499    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-r827w"]
Oct 30 05:52:49.871457 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:52:49.871424    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc8175e4_b674_4586_8e97_e6fc60ef5480.slice/crio-07c932aa1d9134313bb61f00d1eb82ec8ddbac5d078fd79f349902aefa623491 WatchSource:0}: Error finding container 07c932aa1d9134313bb61f00d1eb82ec8ddbac5d078fd79f349902aefa623491: Status 404 returned error can't find the container with id 07c932aa1d9134313bb61f00d1eb82ec8ddbac5d078fd79f349902aefa623491
Oct 30 05:52:50.741446 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:50.741416    5898 generic.go:356] "Generic (PLEG): container finished" podID="cc8175e4-b674-4586-8e97-e6fc60ef5480" containerID="2de2e80796f6c6dff59078d42a1fbb8cf3eabef6da6d64a2f5772fded5d7554a" exitCode=0
Oct 30 05:52:50.741790 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:50.741503    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-r827w" event={"ID":"cc8175e4-b674-4586-8e97-e6fc60ef5480","Type":"ContainerDied","Data":"2de2e80796f6c6dff59078d42a1fbb8cf3eabef6da6d64a2f5772fded5d7554a"}
Oct 30 05:52:50.741790 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:50.741551    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-r827w" event={"ID":"cc8175e4-b674-4586-8e97-e6fc60ef5480","Type":"ContainerStarted","Data":"07c932aa1d9134313bb61f00d1eb82ec8ddbac5d078fd79f349902aefa623491"}
Oct 30 05:52:51.879879 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:51.879854    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-r827w"
Oct 30 05:52:51.962793 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:51.962757    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc8175e4-b674-4586-8e97-e6fc60ef5480-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc8175e4-b674-4586-8e97-e6fc60ef5480-maas-api-openshift-ca\") pod \"cc8175e4-b674-4586-8e97-e6fc60ef5480\" (UID: \"cc8175e4-b674-4586-8e97-e6fc60ef5480\") "
Oct 30 05:52:51.962994 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:51.962865    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc8175e4-b674-4586-8e97-e6fc60ef5480-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc8175e4-b674-4586-8e97-e6fc60ef5480-maas-api-xks-ca\") pod \"cc8175e4-b674-4586-8e97-e6fc60ef5480\" (UID: \"cc8175e4-b674-4586-8e97-e6fc60ef5480\") "
Oct 30 05:52:51.962994 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:51.962969    5898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc8175e4-b674-4586-8e97-e6fc60ef5480-kube-api-access-gf2cb\" (UniqueName: \"kubernetes.io/projected/cc8175e4-b674-4586-8e97-e6fc60ef5480-kube-api-access-gf2cb\") pod \"cc8175e4-b674-4586-8e97-e6fc60ef5480\" (UID: \"cc8175e4-b674-4586-8e97-e6fc60ef5480\") "
Oct 30 05:52:51.963231 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:51.963210    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc8175e4-b674-4586-8e97-e6fc60ef5480-maas-api-openshift-ca" pod "cc8175e4-b674-4586-8e97-e6fc60ef5480" (UID: "cc8175e4-b674-4586-8e97-e6fc60ef5480"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:52:51.963413 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:51.963396    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc8175e4-b674-4586-8e97-e6fc60ef5480-maas-api-openshift-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:52:51.964956 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:51.964932    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc8175e4-b674-4586-8e97-e6fc60ef5480-maas-api-xks-ca" pod "cc8175e4-b674-4586-8e97-e6fc60ef5480" (UID: "cc8175e4-b674-4586-8e97-e6fc60ef5480"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:52:51.965053 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:51.965014    5898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc8175e4-b674-4586-8e97-e6fc60ef5480-kube-api-access-gf2cb" pod "cc8175e4-b674-4586-8e97-e6fc60ef5480" (UID: "cc8175e4-b674-4586-8e97-e6fc60ef5480"). InnerVolumeSpecName "kube-api-access-gf2cb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:52:52.064537 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:52.064473    5898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc8175e4-b674-4586-8e97-e6fc60ef5480-maas-api-xks-ca\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:52:52.064537 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:52.064499    5898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf2cb\" (UniqueName: \"kubernetes.io/projected/cc8175e4-b674-4586-8e97-e6fc60ef5480-kube-api-access-gf2cb\") on node \"ip-10-0-135-98.ec2.internal\" DevicePath \"\""
Oct 30 05:52:52.752541 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:52.752512    5898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-r827w"
Oct 30 05:52:52.752750 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:52.752541    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-r827w" event={"ID":"cc8175e4-b674-4586-8e97-e6fc60ef5480","Type":"ContainerDied","Data":"07c932aa1d9134313bb61f00d1eb82ec8ddbac5d078fd79f349902aefa623491"}
Oct 30 05:52:52.752750 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:52.752583    5898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07c932aa1d9134313bb61f00d1eb82ec8ddbac5d078fd79f349902aefa623491"
Oct 30 05:52:54.499306 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:52:54.499275    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:52:56.817331 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:52:56.817295    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:53:00.385920 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:53:00.385867    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n8njr"]
Oct 30 05:53:11.322070 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:53:11.322032    5898 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-4jgtx_c712ba13-198b-48ba-9464-8b561f465646/manager/0.log"
Oct 30 05:53:11.522793 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:53:11.522763    5898 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-5cfb77bf8c-c28c8_25690daf-01ba-4364-8d5b-10712cf9d193/manager/0.log"
Oct 30 05:53:11.751824 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:53:11.751738    5898 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-9jkvx_62287fbd-b4d1-4b24-9e5f-25aff2c09664/manager/0.log"
Oct 30 05:53:12.371695 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:53:12.371667    5898 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-t2c9k_6a337a14-d243-45dc-8131-0d00b38564e8/server/0.log"
Oct 30 05:53:13.319154 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:53:13.319126    5898 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8484dd48c9-sxchv_b85ad37a-53e3-4c9f-abab-7231172266fb/copy-manifests/0.log"
Oct 30 05:53:13.347141 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:53:13.347113    5898 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8484dd48c9-sxchv_b85ad37a-53e3-4c9f-abab-7231172266fb/manager/0.log"
Oct 30 05:53:14.932606 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:53:14.932574    5898 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-cnvz8_a336d4a9-4960-473b-ac17-1359040170af/revoke-keys/0.log"
Oct 30 05:53:15.677526 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:53:15.677498    5898 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-r827w_cc8175e4-b674-4586-8e97-e6fc60ef5480/revoke-keys/0.log"
Oct 30 05:53:16.435687 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:53:16.435659    5898 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-lmtqm_6d0bd3d6-aa1f-40c9-a0ea-e10fe0308a79/revoke-keys/0.log"
Oct 30 05:53:18.830791 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:53:18.830762    5898 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-fgtkv_99ed5b75-d71c-496a-a260-794b59289962/kuadrant-console-plugin/0.log"
Oct 30 05:53:19.276083 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:53:19.276004    5898 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-n8njr_50ad4653-5ecb-4f30-9533-2ba45e1fa1cd/limitador/0.log"
Oct 30 05:53:20.258757 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:53:20.258726    5898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-67df64d57-nwq4j_d5db038b-f1f8-434b-bd6d-48707dce8b5c/istio-proxy/0.log"
Oct 30 05:53:20.443772 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:53:20.443742    5898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-ft7s9_f7b35ef1-25f4-46ac-9a62-fe5e2e027af4/discovery/0.log"
Oct 30 05:53:21.426792 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:53:21.426764    5898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x_396ecffe-9a19-487d-a9cd-5d20d1b65fcf/istio-proxy/0.log"
Oct 30 05:53:22.389753 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:53:22.389728    5898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5_b79b4fcb-1af2-424b-9b56-f6f9d40f88b2/istio-proxy/1.log"
Oct 30 05:53:22.789463 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:53:22.789392    5898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb_0b40b768-9853-4acd-bf54-9d90b9f5a0ab/istio-proxy/0.log"
Oct 30 05:53:23.842213 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:53:23.842182    5898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5_b79b4fcb-1af2-424b-9b56-f6f9d40f88b2/istio-proxy/0.log"
Oct 30 05:53:26.817258 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:53:26.817222    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:53:35.953167 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:53:35.953140    5898 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-4jgtx_c712ba13-198b-48ba-9464-8b561f465646/manager/0.log"
Oct 30 05:53:36.156496 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:53:36.156467    5898 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-5cfb77bf8c-c28c8_25690daf-01ba-4364-8d5b-10712cf9d193/manager/0.log"
Oct 30 05:53:36.378430 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:53:36.378402    5898 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-9jkvx_62287fbd-b4d1-4b24-9e5f-25aff2c09664/manager/0.log"
Oct 30 05:53:36.855740 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:53:36.855713    5898 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-t2c9k_6a337a14-d243-45dc-8131-0d00b38564e8/server/0.log"
Oct 30 05:53:37.720739 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:53:37.720712    5898 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8484dd48c9-sxchv_b85ad37a-53e3-4c9f-abab-7231172266fb/copy-manifests/0.log"
Oct 30 05:53:37.747576 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:53:37.747543    5898 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8484dd48c9-sxchv_b85ad37a-53e3-4c9f-abab-7231172266fb/manager/0.log"
Oct 30 05:53:39.140625 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:53:39.140593    5898 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-cnvz8_a336d4a9-4960-473b-ac17-1359040170af/revoke-keys/0.log"
Oct 30 05:53:39.885004 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:53:39.884976    5898 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-r827w_cc8175e4-b674-4586-8e97-e6fc60ef5480/revoke-keys/0.log"
Oct 30 05:53:40.626768 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:53:40.626734    5898 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-lmtqm_6d0bd3d6-aa1f-40c9-a0ea-e10fe0308a79/revoke-keys/0.log"
Oct 30 05:53:42.640044 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:53:42.639965    5898 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-fgtkv_99ed5b75-d71c-496a-a260-794b59289962/kuadrant-console-plugin/0.log"
Oct 30 05:53:43.123634 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:53:43.123605    5898 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-n8njr_50ad4653-5ecb-4f30-9533-2ba45e1fa1cd/limitador/0.log"
Oct 30 05:53:43.887545 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:53:43.887515    5898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-67df64d57-nwq4j_d5db038b-f1f8-434b-bd6d-48707dce8b5c/istio-proxy/0.log"
Oct 30 05:53:44.075091 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:53:44.075061    5898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-ft7s9_f7b35ef1-25f4-46ac-9a62-fe5e2e027af4/discovery/0.log"
Oct 30 05:53:45.044573 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:53:45.044541    5898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x_396ecffe-9a19-487d-a9cd-5d20d1b65fcf/istio-proxy/0.log"
Oct 30 05:53:45.909619 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:53:45.909582    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:53:46.034523 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:53:46.034494    5898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5_b79b4fcb-1af2-424b-9b56-f6f9d40f88b2/istio-proxy/1.log"
Oct 30 05:53:46.424367 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:53:46.424336    5898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb_0b40b768-9853-4acd-bf54-9d90b9f5a0ab/istio-proxy/0.log"
Oct 30 05:53:56.817687 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:53:56.817645    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:54:26.818008 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:54:26.817971    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:54:46.051152 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:54:46.051099    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:54:56.817303 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:54:56.817262    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:55:01.452757 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:55:01.452720    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:01.548633 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:55:01.548602    5898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log" limit=1024
Oct 30 05:55:06.478294 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:55:06.478262    5898 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-tt5mj_41a4ec4b-b407-4316-9238-dee6be432d3c/global-pull-secret-syncer/0.log"
Oct 30 05:55:07.126824 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:55:07.126791    5898 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-6bpl4_73a78a0d-d19e-4a96-b8ce-253d631da703/konnectivity-agent/0.log"
Oct 30 05:55:07.665083 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:55:07.665055    5898 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-98.ec2.internal_38fe9c19c3aaab70dd96caa47ec7e827/haproxy/0.log"
Oct 30 05:55:14.327233 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:55:14.327150    5898 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-fgtkv_99ed5b75-d71c-496a-a260-794b59289962/kuadrant-console-plugin/0.log"
Oct 30 05:55:14.581654 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:55:14.581573    5898 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-n8njr_50ad4653-5ecb-4f30-9533-2ba45e1fa1cd/limitador/0.log"
Oct 30 05:55:16.203743 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:55:16.203704    5898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rzzpg/perf-node-gather-daemonset-pxrqx"]
Oct 30 05:55:16.204184 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:55:16.204113    5898 cpu_manager.go:414] "containerMap: removing container" podUID="cc8175e4-b674-4586-8e97-e6fc60ef5480" containerName="revoke-keys"
Oct 30 05:55:16.204184 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:55:16.204126    5898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc8175e4-b674-4586-8e97-e6fc60ef5480" containerName="revoke-keys"
Oct 30 05:55:16.204262 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:55:16.204206    5898 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc8175e4-b674-4586-8e97-e6fc60ef5480" containerName="revoke-keys"
Oct 30 05:55:16.207185 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:55:16.207169    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rzzpg/perf-node-gather-daemonset-pxrqx"
Oct 30 05:55:16.227827 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:55:16.227789    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rzzpg/perf-node-gather-daemonset-pxrqx"]
Oct 30 05:55:16.311191 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:55:16.311163    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rzzpg\"/\"openshift-service-ca.crt\""
Oct 30 05:55:16.311191 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:55:16.311169    5898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rzzpg\"/\"kube-root-ca.crt\""
Oct 30 05:55:16.311394 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:55:16.311236    5898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rzzpg\"/\"default-dockercfg-jv58p\""
Oct 30 05:55:16.319586 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:55:16.319545    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/06b61889-2eec-42b4-923d-2feaceaae89d-lib-modules\") pod \"perf-node-gather-daemonset-pxrqx\" (UID: \"06b61889-2eec-42b4-923d-2feaceaae89d\") " pod="openshift-must-gather-rzzpg/perf-node-gather-daemonset-pxrqx"
Oct 30 05:55:16.319681 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:55:16.319607    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwt7v\" (UniqueName: \"kubernetes.io/projected/06b61889-2eec-42b4-923d-2feaceaae89d-kube-api-access-dwt7v\") pod \"perf-node-gather-daemonset-pxrqx\" (UID: \"06b61889-2eec-42b4-923d-2feaceaae89d\") " pod="openshift-must-gather-rzzpg/perf-node-gather-daemonset-pxrqx"
Oct 30 05:55:16.319720 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:55:16.319693    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/06b61889-2eec-42b4-923d-2feaceaae89d-podres\") pod \"perf-node-gather-daemonset-pxrqx\" (UID: \"06b61889-2eec-42b4-923d-2feaceaae89d\") " pod="openshift-must-gather-rzzpg/perf-node-gather-daemonset-pxrqx"
Oct 30 05:55:16.319756 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:55:16.319718    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/06b61889-2eec-42b4-923d-2feaceaae89d-proc\") pod \"perf-node-gather-daemonset-pxrqx\" (UID: \"06b61889-2eec-42b4-923d-2feaceaae89d\") " pod="openshift-must-gather-rzzpg/perf-node-gather-daemonset-pxrqx"
Oct 30 05:55:16.319793 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:55:16.319755    5898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/06b61889-2eec-42b4-923d-2feaceaae89d-sys\") pod \"perf-node-gather-daemonset-pxrqx\" (UID: \"06b61889-2eec-42b4-923d-2feaceaae89d\") " pod="openshift-must-gather-rzzpg/perf-node-gather-daemonset-pxrqx"
Oct 30 05:55:16.421186 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:55:16.421151    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/06b61889-2eec-42b4-923d-2feaceaae89d-lib-modules\") pod \"perf-node-gather-daemonset-pxrqx\" (UID: \"06b61889-2eec-42b4-923d-2feaceaae89d\") " pod="openshift-must-gather-rzzpg/perf-node-gather-daemonset-pxrqx"
Oct 30 05:55:16.421361 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:55:16.421202    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwt7v\" (UniqueName: \"kubernetes.io/projected/06b61889-2eec-42b4-923d-2feaceaae89d-kube-api-access-dwt7v\") pod \"perf-node-gather-daemonset-pxrqx\" (UID: \"06b61889-2eec-42b4-923d-2feaceaae89d\") " pod="openshift-must-gather-rzzpg/perf-node-gather-daemonset-pxrqx"
Oct 30 05:55:16.421361 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:55:16.421254    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/06b61889-2eec-42b4-923d-2feaceaae89d-podres\") pod \"perf-node-gather-daemonset-pxrqx\" (UID: \"06b61889-2eec-42b4-923d-2feaceaae89d\") " pod="openshift-must-gather-rzzpg/perf-node-gather-daemonset-pxrqx"
Oct 30 05:55:16.421361 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:55:16.421282    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/06b61889-2eec-42b4-923d-2feaceaae89d-proc\") pod \"perf-node-gather-daemonset-pxrqx\" (UID: \"06b61889-2eec-42b4-923d-2feaceaae89d\") " pod="openshift-must-gather-rzzpg/perf-node-gather-daemonset-pxrqx"
Oct 30 05:55:16.421361 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:55:16.421294    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/06b61889-2eec-42b4-923d-2feaceaae89d-lib-modules\") pod \"perf-node-gather-daemonset-pxrqx\" (UID: \"06b61889-2eec-42b4-923d-2feaceaae89d\") " pod="openshift-must-gather-rzzpg/perf-node-gather-daemonset-pxrqx"
Oct 30 05:55:16.421361 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:55:16.421329    5898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/06b61889-2eec-42b4-923d-2feaceaae89d-sys\") pod \"perf-node-gather-daemonset-pxrqx\" (UID: \"06b61889-2eec-42b4-923d-2feaceaae89d\") " pod="openshift-must-gather-rzzpg/perf-node-gather-daemonset-pxrqx"
Oct 30 05:55:16.421361 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:55:16.421358    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/06b61889-2eec-42b4-923d-2feaceaae89d-proc\") pod \"perf-node-gather-daemonset-pxrqx\" (UID: \"06b61889-2eec-42b4-923d-2feaceaae89d\") " pod="openshift-must-gather-rzzpg/perf-node-gather-daemonset-pxrqx"
Oct 30 05:55:16.421628 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:55:16.421362    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/06b61889-2eec-42b4-923d-2feaceaae89d-podres\") pod \"perf-node-gather-daemonset-pxrqx\" (UID: \"06b61889-2eec-42b4-923d-2feaceaae89d\") " pod="openshift-must-gather-rzzpg/perf-node-gather-daemonset-pxrqx"
Oct 30 05:55:16.421628 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:55:16.421451    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/06b61889-2eec-42b4-923d-2feaceaae89d-sys\") pod \"perf-node-gather-daemonset-pxrqx\" (UID: \"06b61889-2eec-42b4-923d-2feaceaae89d\") " pod="openshift-must-gather-rzzpg/perf-node-gather-daemonset-pxrqx"
Oct 30 05:55:16.441271 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:55:16.441244    5898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwt7v\" (UniqueName: \"kubernetes.io/projected/06b61889-2eec-42b4-923d-2feaceaae89d-kube-api-access-dwt7v\") pod \"perf-node-gather-daemonset-pxrqx\" (UID: \"06b61889-2eec-42b4-923d-2feaceaae89d\") " pod="openshift-must-gather-rzzpg/perf-node-gather-daemonset-pxrqx"
Oct 30 05:55:16.520009 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:55:16.519932    5898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rzzpg/perf-node-gather-daemonset-pxrqx"
Oct 30 05:55:16.814016 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:55:16.813990    5898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rzzpg/perf-node-gather-daemonset-pxrqx"]
Oct 30 05:55:16.815369 ip-10-0-135-98 kubenswrapper[5898]: W1030 05:55:16.815343    5898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod06b61889_2eec_42b4_923d_2feaceaae89d.slice/crio-9ab215767e2000113574e49eca7cc27b0d3eb469b8aab31e5eb3b9882d20c209 WatchSource:0}: Error finding container 9ab215767e2000113574e49eca7cc27b0d3eb469b8aab31e5eb3b9882d20c209: Status 404 returned error can't find the container with id 9ab215767e2000113574e49eca7cc27b0d3eb469b8aab31e5eb3b9882d20c209
Oct 30 05:55:17.471577 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:55:17.471527    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rzzpg/perf-node-gather-daemonset-pxrqx" event={"ID":"06b61889-2eec-42b4-923d-2feaceaae89d","Type":"ContainerStarted","Data":"b9c6b8be0e12446f4863df6d8bafda41e270708744eed6f58f6153f901c2a3af"}
Oct 30 05:55:17.472147 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:55:17.471739    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rzzpg/perf-node-gather-daemonset-pxrqx"
Oct 30 05:55:17.472147 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:55:17.471791    5898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rzzpg/perf-node-gather-daemonset-pxrqx" event={"ID":"06b61889-2eec-42b4-923d-2feaceaae89d","Type":"ContainerStarted","Data":"9ab215767e2000113574e49eca7cc27b0d3eb469b8aab31e5eb3b9882d20c209"}
Oct 30 05:55:17.509631 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:55:17.509584    5898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-rzzpg/perf-node-gather-daemonset-pxrqx" podStartSLOduration=1.509571974 podStartE2EDuration="1.509571974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:55:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:55:17.508785577 +0000 UTC m=+4246.732605478" watchObservedRunningTime="2026-10-30 05:55:17.509571974 +0000 UTC m=+4246.733391890"
Oct 30 05:55:17.657842 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:55:17.657812    5898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-r5tdl_eddb252a-1213-4eb9-9219-5a0abedf5587/monitoring-plugin/0.log"
Oct 30 05:55:17.780331 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:55:17.780251    5898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9mr8m_76081e78-ba20-4f7c-b825-a9e0da23c511/node-exporter/0.log"
Oct 30 05:55:17.898859 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:55:17.898828    5898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9mr8m_76081e78-ba20-4f7c-b825-a9e0da23c511/kube-rbac-proxy/0.log"
Oct 30 05:55:18.015518 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:55:18.015491    5898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9mr8m_76081e78-ba20-4f7c-b825-a9e0da23c511/init-textfile/0.log"
Oct 30 05:55:20.023617 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:55:20.023585    5898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-4z6wz_3023960e-7dab-48fe-926b-b016f2e2b5b7/prometheus-operator/0.log"
Oct 30 05:55:20.133812 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:55:20.133784    5898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-4z6wz_3023960e-7dab-48fe-926b-b016f2e2b5b7/kube-rbac-proxy/0.log"
Oct 30 05:55:20.731417 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:55:20.731386    5898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-95b457c5b-n92bg_fd380d55-276e-45eb-8191-c5940ed8385b/thanos-query/0.log"
Oct 30 05:55:20.845984 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:55:20.845958    5898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-95b457c5b-n92bg_fd380d55-276e-45eb-8191-c5940ed8385b/kube-rbac-proxy-web/0.log"
Oct 30 05:55:20.961915 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:55:20.961885    5898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-95b457c5b-n92bg_fd380d55-276e-45eb-8191-c5940ed8385b/kube-rbac-proxy/0.log"
Oct 30 05:55:21.081471 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:55:21.081443    5898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-95b457c5b-n92bg_fd380d55-276e-45eb-8191-c5940ed8385b/prom-label-proxy/0.log"
Oct 30 05:55:21.207437 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:55:21.207410    5898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-95b457c5b-n92bg_fd380d55-276e-45eb-8191-c5940ed8385b/kube-rbac-proxy-rules/0.log"
Oct 30 05:55:21.328042 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:55:21.328012    5898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-95b457c5b-n92bg_fd380d55-276e-45eb-8191-c5940ed8385b/kube-rbac-proxy-metrics/0.log"
Oct 30 05:55:23.486295 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:55:23.486269    5898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rzzpg/perf-node-gather-daemonset-pxrqx"
Oct 30 05:55:24.809255 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:55:24.809226    5898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7ddddfc654-x4l4p_1fb50298-92f0-4411-a3dd-ddef516b4314/console/0.log"
Oct 30 05:55:26.816698 ip-10-0-135-98 kubenswrapper[5898]: E1030 05:55:26.816660    5898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-srsjc" containerName="extractor"
Oct 30 05:55:28.861720 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:55:28.861692    5898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bbk54_ddad100f-6a08-4da8-a51a-f3f2fd5438bb/dns/0.log"
Oct 30 05:55:28.975048 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:55:28.975016    5898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bbk54_ddad100f-6a08-4da8-a51a-f3f2fd5438bb/kube-rbac-proxy/0.log"
Oct 30 05:55:29.344460 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:55:29.344432    5898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gndxs_885d476c-7568-4b63-ab9d-1da7608706f5/dns-node-resolver/0.log"
Oct 30 05:55:29.625853 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:55:29.625777    5898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-67df64d57-nwq4j_d5db038b-f1f8-434b-bd6d-48707dce8b5c/istio-proxy/0.log"
Oct 30 05:55:31.170093 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:55:31.170059    5898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zqs6v_daf6535c-7f33-407a-a5d9-e1c9525c8cb8/node-ca/0.log"
Oct 30 05:55:31.579388 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:55:31.579360    5898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-ft7s9_f7b35ef1-25f4-46ac-9a62-fe5e2e027af4/discovery/0.log"
Oct 30 05:55:31.880354 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:55:31.880283    5898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-5ht7x_396ecffe-9a19-487d-a9cd-5d20d1b65fcf/istio-proxy/0.log"
Oct 30 05:55:32.210178 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:55:32.210110    5898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5_b79b4fcb-1af2-424b-9b56-f6f9d40f88b2/istio-proxy/0.log"
Oct 30 05:55:32.211491 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:55:32.211473    5898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-tpdh5_b79b4fcb-1af2-424b-9b56-f6f9d40f88b2/istio-proxy/1.log"
Oct 30 05:55:32.312551 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:55:32.312523    5898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-zmgcb_0b40b768-9853-4acd-bf54-9d90b9f5a0ab/istio-proxy/0.log"
Oct 30 05:55:33.021866 ip-10-0-135-98 kubenswrapper[5898]: I1030 05:55:33.021836    5898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-67df64d57-nwq4j_d5db038b-f1f8-434b-bd6d-48707dce8b5c/istio-proxy/0.log"